Red Hat OpenShift Container Platform

Using the on-cluster image mode to apply a custom layered image

You can use the on-cluster build process to apply a custom layered image to your cluster by creating a MachineOSConfig custom resource (CR).

The MachineOSConfig CR specifies the following parameters:

the Containerfile to build
the machine config pool to associate the build
where the final image should be pushed and pulled from
the push and pull secrets to use

You can create only one MachineOSConfig CR for each machine config pool.

Prerequisites

You have the pull secret in the openshift-machine-config-operator namespace that the Machine Config Operator (MCO) needs in order to pull the base operating system image from your repository. By default, the MCO uses the cluster global pull secret, which it synchronizes into the openshift-machine-config-operator namespace. You can add your pull secret to the Red Hat OpenShift Container Platform global pull secret or you can use a different pull secret. For information on modifying the global pull secret, see "Updating the global cluster pull secret".
You have the push secret of the registry that the MCO needs to push the new custom layered image to. The credentials provided by the secret must also grant permission to delete an image from the registry.

In a disconnected environment, ensure that the disconnected cluster can access the registry where you want to push the image. Image mirroring applies only to pulling images.
You have the pull secret that your nodes need to pull the new custom layered image from your registry. This should be a different secret than the one used to push the image to the repository.
You are familiar with how to configure a Containerfile. Instructions on how to create a Containerfile are beyond the scope of this documentation.
Optional: You have a separate machine config pool for the nodes where you want to apply the custom layered image. One benefit to having a custom machine config pool for the nodes it that you can easily revert to the base image, if needed. For more information, see "Reverting an on-cluster layered node".

Procedure

Create a MachineOSconfig object:
1. Create a YAML file similar to the following:
  apiVersion: machineconfiguration.openshift.io/v1 kind: MachineOSConfig metadata: name: layered-image spec: machineConfigPool: name: layered-image containerFile: - containerfileArch: NoArch content: |- FROM configs AS final RUN dnf install -y cowsay && \ dnf clean all && \ bootc container lint imageBuilder: imageBuilderType: Job baseImagePullSecret: name: global-pull-secret-copy renderedImagePushSpec: image-registry.openshift-image-registry.svc:5000/openshift/os-image:latest renderedImagePushSecret: name: builder-dockercfg-mtcl23
  where:
  
  apiVersion
  
  Specifies the machineconfiguration.openshift.io/v1 API that is required for MachineConfig CRs.
  
  metadata.name
  
  Specifies a name for the MachineOSConfig object. The name must match the name of the associated machine config pool. This name is used with other on-cluster image mode resources. The examples in this documentation use the name layered-image.
  
  spec.machineConfigPool.name
  
  Specifies the name of the machine config pool associated with the nodes where you want to deploy the custom layered image. The examples in this documentation use the layered-image machine config pool.
  
  spec.containerFile
  
  Specifies the Containerfile to configure the custom layered image.
  
  spec.containerFile.containerfileArch
  
  Specifies the architecture this containerfile is to be built for: ARM64, AMD64, PPC64LE, S390X, or NoArch. The default is NoArch, which defines a Containerfile that can be applied to any architecture.
  
  spec.imageBuilder
  
  Specifies the name of the image builder to use. This must be Job, which is a reference to the job object that is managing the image build.
  
  spec.baseImagePullSecret
  
  Specifies the name of the pull secret that the MCO needs to pull the base operating system image from the registry. By default, the global pull secret is used. This parameter is optional.
  
  spec.renderedImagePushSpec
  
  Specifies the image registry to push the newly-built custom layered image to. This can be any registry that your cluster has access to in the host[:port][/namespace]/name or svc_name.namespace.svc[:port]/repository/name:<tag> format. This example uses the internal Red Hat OpenShift Container Platform registry. You can specify a mirror registry if you cluster is properly configured to use a mirror registry.
  
  spec.renderedImagePushSecret
  
  Specifies the name of the push secret that the MCO needs to push the newly-built custom layered image to that registry.
2. Create the MachineOSConfig object:
  $ oc create -f <filename>.yaml
If necessary, when the MachineOSBuild object has been created and is in the READY state, modify the node spec for the nodes where you want to use the new custom layered image:
1. Check that the MachineOSBuild object is ready, by running the following command:
  $ oc get machineosbuild
  When the SUCCEEDED value is True, the build is complete:
  Example output showing that the MachineOSBuild object is ready
  NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED AGE layered-image-ad5a3cad36303c363cf458ab0524e7c0-builder False False True False False 43s
2. Edit the nodes where you want to deploy the custom layered image by adding a label for the machine config pool you specified in the MachineOSConfig object:
  $ oc label node <node_name> 'node-role.kubernetes.io/<mcp_name>='
  where:
  
  node-role.kubernetes.io/<mcp_name>=
  
  Specifies a node selector that identifies the nodes to deploy the custom layered image.
  
  When you save the changes, the MCO drains, cordons, and reboots the nodes. After the reboot, the node uses the new custom layered image.

Verification

Verify that the new pods are ready by running the following command:

$ oc get pods -n openshift-machine-config-operator

Example output

NAME                                                                    READY   STATUS    RESTARTS   AGE
build-layered-image-ad5a3cad36303c363cf458ab0524e7c0-hxrws              2/2     Running   0          2m40s
# ...
machine-os-builder-6fb66cfb99-zcpvq                                     1/1     Running   0          2m42s

The build pod where the custom layered image is building is named in the build-<MachineOSConfig_CR_name>-<hash> format. The machine-os-builder-* pod can be used for troubleshooting.

Verify the custom layered image build by running a command similar to the following:

$ oc get machineconfigpool <mcp_name> -o yaml

Example output

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  labels:
    machineconfiguration.openshift.io/mco-built-in: ""
    pools.operator.machineconfiguration.openshift.io/layered: ""
  name: layered
# ...
status:
# ...
  conditions
# ...
  - lastTransitionTime: "2025-09-09T13:43:35Z"
    message: 'Failed to build OS image for pool worker (MachineOSBuild: worker-2d03dc921ff0c242c5892a3ef1ed1608):
      Failed: Build Failed'
    reason: BuildFailed
    status: "True"
    type: ImageBuildDegraded

The reason: BuildFailed status indicates whether the custom layered image build failed. If False, the build succeeded. If True, the build failed. You can use the build pod logs to troubleshoot any build failures.

Verify the current stage of your layered build by running the following command:

$ oc get machineosbuilds

Example output

NAME                                             PREPARED   BUILDING   SUCCEEDED   INTERRUPTED   FAILED   AGE
layered-image-ad5a3cad36303c363cf458ab0524e7c0   False      True       False       False         False    12m

The MachineOSBuild object is named in the <MachineOSConfig_CR_name>-<hash> format.

The build is complete when BUILDING is False and SUCCEEDED is True.

When the build is complete, verify that the image has been applied to the nodes in the affected pool by running a command similar to the following:

$ oc describe machineconfignode/<machine_config_node_name>

Example machine config node output

Name:         ip-10-0-14-86.us-west-1.compute.internal
API Version:  machineconfiguration.openshift.io/v1
Kind:         MachineConfigNode
# ...
Spec:
  Config Image:
    Desired Image:  image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-image@sha256:b485378fd8f7963ed74f14ce64f4f1e511e1601d49302b3046b1b78a83f539e3
  Config Version:
    Desired:  rendered-worker-d63c7736923b60b8b82492ae9a1eef40
  Node:
    Name:  ip-10-0-14-86.us-west-1.compute.internal
  Pool:
    Name:  worker
# ...
Status:
  Conditions:
# ...
    Message:               Action during update to image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-image@sha256:b485378fd8f7963ed74f14ce64f4f1e511e1601d49302b3046b1b78a83f539e3: Successfully pulled OS image image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-image@sha256:b485378fd8f7963ed74f14ce64f4f1e511e1601d49302b3046b1b78a83f539e3 from registry
    Reason:                ImagePulledFromRegistry
    Status:                False
    Type:                  ImagePulledFromRegistry
# ...
  Config Image:
    Current Image:  image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-image@sha256:b485378fd8f7963ed74f14ce64f4f1e511e1601d49302b3046b1b78a83f539e3
    Desired Image:  image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-image@sha256:b485378fd8f7963ed74f14ce64f4f1e511e1601d49302b3046b1b78a83f539e3
# ...

The digested image pull spec for the new custom layered image is in the Spec.Config Image.Desired Image parameter.

The ImagePulledFromRegistry condition is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Verify that the MachineOSConfig object contains a reference to the new custom layered image by running the following command:

$ oc describe machineosconfig <object_name>

Example digested image pull spec

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineOSConfig
metadata:
  annotations:
    machineconfiguration.openshift.io/current-machine-os-build: layered-9a8f89455246fa0c42ecee6ff1fa1a45
  labels:
    machineconfiguration.openshift.io/createdByOnClusterBuildsHelper: ""
  name: layered-image
# ...
status:
  currentImagePullSpec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-image@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754

where:

status.currentImagePullSpec: Specifies the digested image pull spec for the new custom layered image.

Verify that the appropriate nodes are using the new custom layered image:

Start a debug session as root for a control plane node by running the following command:
```
$ oc debug node/<node_name>
```
Set /host as the root directory within the debug shell:
```
sh-4.4# chroot /host
```

Run the rpm-ostree status command to view that the custom layered image is in use:

sh-5.1# rpm-ostree status

Example output

# ...
Deployments:
* ostree-unverified-registry:image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754
                   Digest: sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754
                  Version: 419.94.202502100215-0 (2025-02-12T19:20:44Z)

The Deployments stanza includes the digested image pull spec for the new custom layered image.