Agenda

• Brief introduction into baremetal inspection
• Ironic Inspector architecture
• Beyond basic inspection:
  • Plugins
  • Introspection rules
  • Hardware discovery
• Looking into the future

Baremetal Inspection

Bare metal inspection (or introspection^[*]) is a process of finding out hardware properties requried for deployment from the node itself.

The bare minimum for inspection in case of Ironic:

• CPU architecture and count
• Memory size
• Hard drive size
• Hardware addresses of NIC's

[*] in this context we use both words and treat them as synonyms

Baremetal Inspection

Out-of-band

Uses vendor BMC features.
Pros: quick and reliable. Cons: highly vendor-specific^[*] with limited customization.

In-band

Works by booting a ramdisk on the node.
Pros: can collect essentially anything from a wide selection of hardware. Cons: slow and more fragile.

Implemented by Ironic Inspector.

[*] The Redfish specification might change this situation.

Ironic Inspector Project

A separate service under the Bare metal project umbrella.

96 commits by 16 contributors from 8 companies.

5 core team members from 4 companies (not counting the main ironic core team).

Architecture

• Ironic Python Agent (IPA) - the generic ramdisk for Ironic
• Static DHCP server (usually dnsmasq) instructing hosts to boot IPA with inspection.
• Ironic Inspector service:
  • Provides HTTP API
  • Manages access to the DHCP server
  • Accepts and processes data from the ramdisk
• The python-ironic-inspector-client project provides CLI and a Python client library.

Flow

1. A node is created in Ironic with its power credentials.
2. In-band inspection is requested for a node via the Ironic Inspector API
3. Ironic Inspector configures firewall to ensure that the node will boot IPA with the correct kernel arguments
4. Ironic Inspector sets the node boot device to network and powers it on
5. IPA boots on the node and collects information
6. IPA posts the collected information to the Ironic Inspector API
7. Ironic Inspector figures out which Ironic node this information belongs to
8. Ironic Inspector processes the information and updates the node record in the Ironic database
9. Ironic Inspector optionally stores the whole processed data in the object storage (Swift).

Beyond Basic Inspection

Plugins: processing hooks

Allows writing Python plugins, to extend both processing stages:

• Checking and preprocessing node information right after receiving it
• Making changes to the Ironic node based on collected data

Example

class RamdiskErrorHook(ironic_inspector.plugins.base.ProcessingHook):
def before_processing(self, introspection_data, **kwargs):
error = introspection_data.get('error')
if error:
    raise RuntimeError(_('Ramdisk reported error: %s') % error)
    

Plugins: collectors

Collectors extend the introspection process at the data collection stage, i.e. during the ramdisk run.

Adding a new collector requires rebuilding the ramdisk image. Enabling an existing collector is possible via the kernel command line.

Example

def collect_cpuinfo(data, failures):
try:
data['cpuinfo'] = open('/proc/cpuinfo', 'rt').read()
except EnvironmentError as exc:
failures.add('Unable to read cpuinfo: %s', exc)
    

Introduced in the Liberty release.

Extra hardware plugin

Collects a huge number of hardware facts from a node.

Requires the hardware Python package on the ramdisk

Enable extra-hardware collector and extra_hardware processing hook to use.

Can also benchmark CPU, main memory and disks. Pass ipa-inspection-benchmarks=cpu,mem,disk to the IPA kernel command line to enable (takes time!).

Introspection rules

Allows a user to define actions to run during the introspection data processing.

Uses a simple JSON-based domain-specific language.

Example

{
"conditions": [
{"op": "lt", "field": "inventory.memory.physical_mb",
 "value": 1024}
],
"actions": [
{"action": "fail", "message": "memory is too low"}
]
}

Introduced in the Liberty release.

Introspection rules

Example: vendor capability

{
"conditions": [
{"op": "contains",
 "field": "inventory.system_vendor.manufacturer",
 "value": "Awesome Inc"},
{"op": "eq",
 "field": "inventory.disks[*].rotational",
 "value": false,
 "multiple": "all"}
],
"actions": [
{"action": "set-capability", "name": "awesome-ssd",
 "value": "true"}
]
}

Sets a given capability for nodes of a specific vendor with all disks not rotational.

Hardware Discovery

Enrolling of new hardware combined with its inspection.

Enabled by setting the node_not_found_hook configuration option to enroll.

Flow

1. A new node is powered on manually.
2. IPA is booted, data is posted to Ironic Inspector.
3. If Ironic Inspector cannot find the node in Ironic, it creates a new one, then proceeds with processing.

Introspection rules can be used to populate power credentials

Introduced in the Mitaka release.

Hardware Discovery

Example Rules

{
"conditions": [
{"op": "contains",
 "field": "inventory.system_vendor.manufacturer",
 "value": "Dell"},
{"op": "eq", "field": "auto_discovered", "value": true}
],
"actions": [
{"action": "set-attribute", "path": "/driver",
 "value": "pxe_drac"},
{"action": "set-attribute", "path": "/driver_info/drac_host",
 "value": "{data[inventory][bmc_address]}"},
{"action": "set-attribute", "path": "/driver_info/drac_username",
 "value": "root"},
{"action": "set-attribute", "path": "/driver_info/drac_password",
 "value": "calvin"},
]
}

Future Plans

• HA architecture
• CMDB integration
• Further development of introspection rules

Links

• PyPI: https://pypi.python.org/pypi/ironic-inspector
• Bugs: https://bugs.launchpad.net/ironic-inspector
• https://dtantsur.github.io/talks/mitaka-inspector-status

Thank you!