Red Hat OpenShift Container Platform

About on-cluster image mode

You can use the image mode for OpenShift on-cluster build process to apply a custom layered image to your nodes by creating a MachineOSConfig custom resource (CR).

When you create the object, the Machine Config Operator (MCO) creates a MachineOSBuild object and a builder pod. The process also creates transient objects, such as config maps, which are cleaned up after the build is complete. The MachineOSBuild object and the associated builder-* pod use the same naming scheme, <MachineOSConfig_CR_name>-<hash>, for example:

Example MachineOSBuild object

NAME                                             PREPARED   BUILDING   SUCCEEDED   INTERRUPTED   FAILED
layered-image-c8765e26ebc87e1e17a7d6e0a78e8bae   False      False      True        False         False

Example builder pod

NAME                                                      READY   STATUS      RESTARTS        AGE
build-layered-image-c8765e26ebc87e1e17a7d6e0a78e8bae      2/2     Running     0               11m

You should not need to interact with these new objects or the machine-os-builder pod. However, you can use all of these resources for troubleshooting, if necessary.

When the build is complete, the MCO pushes the new custom layered image to your repository and rolls the image out to the nodes in the associated machine config pool. You can see the digested image pull spec for the new custom layered image in the MachineOSConfig object. This is now the active image pull spec for this MachineOSConfig.

Example digested image pull spec

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineOSConfig
metadata:
  annotations:
    machineconfiguration.openshift.io/current-machine-os-build: layered-9a8f89455246fa0c42ecee6ff1fa1a45
  labels:
    machineconfiguration.openshift.io/createdByOnClusterBuildsHelper: ""
  name: layered-image
# ...
status:
  currentImagePullSpec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-image@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754

You can test a MachineOSBuild object to make sure it builds correctly without rolling out the custom layered image to active nodes by using a custom machine config pool that contains non-production nodes. Alternatively, you can use a custom machine config pool that has no nodes. The MachineOSBuild object builds even if there are no nodes for the MCO to deploy the custom layered image onto.

You can apply a custom layered image to any machine config pool in your cluster, including the control plane, worker, or custom pools.

For single-node OpenShift clusters, you can apply a custom layered image to the control plane node only.

Making certain changes to a MachineOSConfig object triggers an automatic rebuild of the associated custom layered image. You can mitigate the effects of the rebuild by pausing the machine config pool where the custom layered image is applied as described in "Pausing the machine config pools". While the pools are paused, the MCO does not roll out the newly built image to the nodes after the build is complete. However, the build runs regardless of whether the pool is paused or not. For example, if you want to remove and replace a MachineOSCOnfig object, pausing the machine config pools before making the change prevents the MCO from reverting the associated nodes to the base image, reducing the number of reboots needed.

When a machine config pool is paused, the oc get machineconfigpools reports the following status:

Example output

NAME      CONFIG                                              UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master    rendered-master-a0b404d061a6183cc36d302363422aba    True      False      False      3              3                   3                     0                      4h14m
worker    rendered-worker-221507009cbcdec0eec8ab3ccd789d18    False     False      False      2              2                   2                     0                      4h14m

The worker machine config pool is paused, as indicated by the three False statuses and the READYMACHINECOUNT at 0.

After the changes have been rolled out, you can unpause the machine config pool.

In the case of a build failure, for example due to network issues or an invalid secret, the MCO retries the build three additional times before the job fails. The MCO creates a different build pod for each build attempt. Note that the MCO automatically removes these build pods after a short period of time. Also, the affected machine config pool reports a build failure through the ImageBuildDegraded status condition. You can use the build pod logs to troubleshoot any build failures.

Example failed MachineOSBuild object

NAME                                             PREPARED   BUILDING   SUCCEEDED   INTERRUPTED   FAILED   AGE
layered-image-c8765e26ebc87e1e17a7d6e0a78e8bae   False      False      False        False        True     12m

You can manually rebuild your custom layered image by either modifying your MachineOSConfig object or applying an annotation to the MachineOSConfig object. For more information, see "Rebuilding an on-cluster custom layered image".

If you used a custom machine config pool to apply an on-cluster layered image to a node, you can remove the custom layered image from the node and revert to the base image. For more information, see "Reverting an on-cluster layered node".

You can modify an on-custom layered image as needed, to install additional packages, remove existing packages, change repositories, update secrets, or other similar changes, by editing the MachineOSConfig object. For more information, see "Modifying a custom layered image".