About machine health checks on bare metal

Machine deletion on bare metal cluster triggers reprovisioning of a bare metal host. Usually bare metal reprovisioning is a lengthy process, during which the cluster is missing compute resources and applications might be interrupted.

There are two ways to change the default remediation process from machine deletion to host power-cycle:

  1. Annotate the MachineHealthCheck resource with the machine.openshift.io/remediation-strategy: external-baremetal annotation.

  2. Create a Metal3RemediationTemplate resource, and refer to it in the spec.remediationTemplate of the MachineHealthCheck.

After using one of these methods, unhealthy machines are power-cycled by using Baseboard Management Controller (BMC) credentials.

About the annotation-based remediation process

The annotation-based remediation process performs the following steps:

  1. The MachineHealthCheck (MHC) controller detects that a node is unhealthy.

  2. The MHC notifies the bare metal machine controller which requests to power-off the unhealthy node.

  3. After the power is off, the node is deleted, which allows the cluster to reschedule the affected workload on other nodes.

  4. The bare metal machine controller requests to power on the node.

  5. After the node is up, the node re-registers itself with the cluster, resulting in the creation of a new node.

  6. After the node is recreated, the bare metal machine controller restores the annotations and labels that existed on the unhealthy node before its deletion.

Note

If the power operations did not complete, the bare metal machine controller triggers the reprovisioning of the unhealthy node unless this is a control plane node or a node that was provisioned externally.
About the metal3-based remediation process

The metal3-based remediation process performs the following steps:

  1. The MachineHealthCheck (MHC) controller detects that a node is unhealthy.

  2. The MHC creates a metal3 remediation custom resource for the metal3 remediation controller, which requests to power-off the unhealthy node.

  3. After the power is off, the node is deleted, which allows the cluster to reschedule the affected workload on other nodes.

  4. The metal3 remediation controller requests to power on the node.

  5. After the node is up, the node re-registers itself with the cluster, resulting in the creation of a new node.

  6. After the node is recreated, the metal3 remediation controller restores the annotations and labels that existed on the unhealthy node before its deletion.

Note

If the power operations did not complete, the metal3 remediation controller triggers the reprovisioning of the unhealthy node unless this is a control plane node or a node that was provisioned externally.

↑ Up: About power-based remediation of bare metal | ⌂ Home: Red Hat OpenShift Container Platform | Next: Unloading the kernel module →