Red Hat build of MicroShift

Install

Mirror container images into an existing registry

Using a custom air-gapped container registry, or mirror, is necessary with certain user environments and workload requirements. Mirroring allows for the transfer of container images and updates to air-gapped environments where they can be installed on a {microshift-short} instance.

To create an air-gapped mirror registry for {microshift-short} containers, you must complete the following steps:

Get the container image list to be mirrored.
Configure the mirroring prerequisites, including secure signatures management.
Download images on a host with internet access.
Copy the downloaded image directory to an air-gapped site.
Upload images to a mirror registry in an air-gapped site.
Configure your {microshift-short} hosts to use the mirror registry.

Getting the mirror registry container image list

To use a mirror registry, you must know which container image references are used by a specific version of {microshift-short}. These references are provided in the release-<arch>.json files that are part of the microshift-release-info RPM package.

To mirror the Operator Lifecycle Manager (OLM) in disconnected environments, add the references provided in the release-olm-$ARCH.json that is included in the microshift-olm RPM and follow the same procedure. Use the oc-mirror CLI plugin for mirroring Operator catalogs and Operators.

Prerequisites

You have installed jq.

Procedure

Access the list of container image references by using one of the following methods:
- If the package is installed on the {microshift-short} host, get the location of the files by running the following command:
  $ rpm -ql microshift-release-info
  Example output
  /usr/share/microshift/release/release-x86_64.json
- If the package is not installed on a {microshift-short} host, download and unpack the RPM package without installing it by running the following command:
  $ rpm2cpio microshift-release-info*.noarch.rpm | cpio -idmv
  Example output
  /usr/share/microshift/release/release-x86_64.json

Extract the list of container images into the microshift-container-refs.txt file by running the following commands:

$ RELEASE_FILE=/usr/share/microshift/release/release-$(uname -m).json

$ jq -r '.images | .[]' ${RELEASE_FILE} > microshift-container-refs.txt

After the microshift-container-refs.txt file is created with the {microshift-short} container image list, you can append the file with other user-specific image references before running the mirroring procedure.

Configuring mirroring prerequisites

You must create a container image registry credentials file that allows the mirroring of images from your internet-connected mirror host to your air-gapped mirror. Follow the instructions in the "Configuring credentials that allow images to be mirrored" link provided in the "Additional resources" section. These instructions guide you to create a ~/.pull-secret-mirror.json file on the mirror registry host that includes the user credentials for accessing the mirror.

Example mirror registry pull secret entry

In this example, the following section is added to the pull secret file for the microshift_quay:8443 mirror registry by using microshift:microshift as username and password.

Example mirror registry section for pull secret file

"<microshift_quay:8443>": {
    "auth": "<microshift_auth>",
    "email": "<microshift_quay@example.com>"
},

Replace the <registry_host>:<port> value microshift_quay:8443 with the hostname and port of your mirror registry server.
Replace the <microshift_auth> value with the user password.
Replace the </microshift_quay@example.com> value with the user email.

Downloading container images

After you have located the container list and completed the mirroring prerequisites, download the container images to a host with internet access.

Prerequisites

You logged into a host with access to the internet.
The .pull-secret-mirror.json file and microshift-containers directory contents are available locally.

Procedure

Install the skopeo tool used for copying the container images by running the following command:
```
$ sudo dnf install -y skopeo
```
Set the environment variable that points to the pull secret file:
```
$ PULL_SECRET_FILE=~/.pull-secret-mirror.json
```
Set the environment variable that points to the list of container images:
```
$ IMAGE_LIST_FILE=~/microshift-container-refs.txt
```
Set the environment variable that points to the destination directory for storing the downloaded data:
```
$ IMAGE_LOCAL_DIR=~/microshift-containers
```

Run the following script to download the container images to the ${IMAGE_LOCAL_DIR} directory:

while read -r src_img ; do
   # Remove the source registry prefix
   dst_img=$(echo "${src_img}" | cut -d '/' -f 2-)

   # Run the image download command
   echo "Downloading '${src_img}' to '${IMAGE_LOCAL_DIR}'"
   mkdir -p "${IMAGE_LOCAL_DIR}/${dst_img}"
   skopeo copy --all --quiet \
      --preserve-digests \
      --authfile "${PULL_SECRET_FILE}" \
      docker://"${src_img}" dir://"${IMAGE_LOCAL_DIR}/${dst_img}"

done < "${IMAGE_LIST_FILE}"

Uploading container images to a mirror registry

To use your container images at an air-gapped site, upload them to the mirror registry by using the following procedure.

Prerequisites

You logged into a host with access to microshift-quay.
The .pull-secret-mirror.json file is available locally.
The microshift-containers directory contents are available locally.

Procedure

Install the skopeo tool used for copying the container images by running the following command:
```
$ sudo dnf install -y skopeo
```
Set the environment variables pointing to the pull secret file:
```
$ IMAGE_PULL_FILE=~/.pull-secret-mirror.json
```
Set the environment variables pointing to the local container image directory:
```
$ IMAGE_LOCAL_DIR=~/microshift-containers
```
Set the environment variables pointing to the mirror registry URL for uploading the container images:
```
$ TARGET_REGISTRY=<registry_host>:<port>
```
- Replace <registry_host>:<port> with the hostname and port of your mirror registry server.

Run the following script to upload the container images to the ${TARGET_REGISTRY} mirror registry:

pushd "${IMAGE_LOCAL_DIR}" >/dev/null
while read -r src_manifest ; do
  local src_img
  src_img=$(dirname "${src_manifest}")
  # Add the target registry prefix and remove SHA
  local -r dst_img="${TARGET_REGISTRY}/${src_img}"
  local -r dst_img_no_tag="${TARGET_REGISTRY}/${src_img%%[@:]*}"
  # Run the image upload
  echo "Uploading '${src_img}' to '${dst_img}'"
  skopeo copy --all --quiet \
     --preserve-digests \
     --authfile "${IMAGE_PULL_FILE}" \
     dir://"${IMAGE_LOCAL_DIR}/${src_img}" docker://"${dst_img}"
done < <(find . -type f -name manifest.json -printf '%P\n')
popd >/dev/null

Configuring hosts for mirror registry access

To configure a {microshift-short} host to use a mirror registry, you must give the {microshift-short} host access to the registry by creating a configuration file that maps the Red Hat registry host names to the mirror.

Prerequisites

Your mirror host has access to the internet.
The mirror host can access the mirror registry.
You configured the mirror registry for use in your restricted network.
You downloaded the pull secret and modified it to include authentication to your mirror repository.

Procedure

Log in to your {microshift-short} host.
Enable the SSL certificate trust on any host accessing the mirror registry by completing the following steps:
1. Copy the rootCA.pem file from the mirror registry, for example, <registry_path>/quay-rootCA, to the {microshift-short} host at the /etc/pki/ca-trust/source/anchors directory.
2. Enable the certificate in the system-wide truststore configuration by running the following command:
  $ sudo update-ca-trust

Create the /etc/containers/registries.conf.d/999-microshift-mirror.conf configuration file that maps the Red Hat registry host names to the mirror registry:

Example mirror configuration file

[[registry]]
    prefix = ""
    location = "<registry_host>:<port>"
    mirror-by-digest-only = true
    insecure = false

[[registry]]
    prefix = ""
    location = "quay.io"
    mirror-by-digest-only = true
[[registry.mirror]]
    location = "<registry_host>:<port>"
    insecure = false

[[registry]]
    prefix = ""
    location = "registry.redhat.io"
    mirror-by-digest-only = true
[[registry.mirror]]
    location = "<registry_host>:<port>"
    insecure = false

[[registry]]
    prefix = ""
    location = "registry.access.redhat.com"
    mirror-by-digest-only = true
[[registry.mirror]]
    location = "<registry_host>:<port>"
    insecure = false

Replace <registry_host>:<port> with the hostname and port of your mirror registry server, for example, <microshift-quay:8443>.

Enable the {microshift-short} service by running the following command:
```
$ sudo systemctl enable microshift
```
Reboot the host by running the following command:
```
$ sudo reboot
```

About image mode for Red Hat Enterprise Linux (RHEL)

By using image mode for RHEL, you can use the same tools and techniques for the operating system that you use with application containers. Image mode for RHEL is a deployment method that uses a container-native approach to build, deploy, and manage the operating system as a rhel-bootc image.

This container image uses standard OCI or Docker containers as a transport and delivery format for base operating system updates.
A bootc image includes a Linux kernel that is used to start the operating system.
By using bootc containers, developers, operations administrators, and solution providers can all use the same container-native tools and techniques.

Image mode for RHEL splits the creation and installation of software changes into two steps: one on a build system and one on a running target system.

In the build-system step, a Podman build inspects the RPM files available for installation, determines any dependencies, and creates an ordered list of chained steps to complete. Along with any other system configuration steps taking place, the end result is a new operating system available to install.
In the running-target-system step, a bootc update downloads, unpacks, and prepares the new operating system to be started alongside the currently running system. Local configuration changes are carried forward to the new operating system. These changes take effect only when the system is restarted and the new operating system image replaces the previously running one.

Preparing for bootc image building

Use the image builder tool to compose customized {microshift-short} bootc images optimized for edge deployments. You can run a {microshift-short} node with your applications on a {op-system-image} virtual machine for development and testing first, then use your whole solution in edge production environments.

Use the following RHEL documentation to understand the full details of using {op-system-image}:

Follow the instructions at the following link:
- Using image mode for RHEL to build, deploy, and manage operating systems

For offline or disconnected configurations, embed all container image dependencies as part of the system image. When container images are downloaded without being embedded into the system image, CRI-O wipes them off during unclean shutdowns, such as when a power loss occurs. In this case, you can only restore those container images when the system is online.

The image mode for RHEL with {microshift-short} workflow

Before you can use {op-system-image} with {microshift-short}, you must verify the availability of required resources.

Required resources

Ensure that the following resources are available:

A RHEL 9.x host with an active Red Hat subscription for building {microshift-short} bootc images.
A remote registry for storing and accessing rhel-bootc images.
An AArch64 or x86_64 system architecture.

Workflow

The workflow for using {op-system-image} with {microshift-short} includes the following steps:

Find and use a prebuilt {microshift-short} container image to install RHEL.
If the prebuilt {microshift-short} container image requires customization, build a custom {microshift-short} container image.
Run the container image.

The rpm-ostree file system used by {op-system-ostree} is not supported in {op-system-image}. Do not use the rpm-ostree file system to modify deployments that use {op-system-image}.

Getting the published bootc image for {microshift-short}

You can use the {microshift-short} container images to install {op-system-image}.

Prerequisites

You have an x86_64 or AArch64 platform.
You have access to the registry.redhat.io registry.

Procedure

Navigate to the Red Hat Ecosystem Catalog.
Search for the {microshift-short} container image by using the microshift-bootc keyword.
Open the container image page of the {microshift-short} container image.
Select the Get this image tab to view instructions for downloading the image.
Get access to the latest image on x86_64 and AArch64 platforms by logging into the registry using the following command:
```
$ sudo podman login registry.redhat.io
```

Download the bootc image by running the following command:

$ podman pull registry.redhat.io/openshift4/microshift-bootc-rhel{op-system-version-major}:v{product-version}

Building the bootc image

Build your Red Hat Enterprise Linux (RHEL) that contains {microshift-short} as a bootable container image by using a Containerfile.

Prerequisites

A RHEL 9.x host with an active Red Hat subscription for building {microshift-short} bootc images and running containers.
You logged into the RHEL 9.x host by using the user credentials that have sudo permissions.
The rhocp and fast-datapath repositories are accessible in the host subscription. The repositories do not necessarily need to be enabled on the host.
You have a remote registry such as Red Hat Quay for storing and accessing bootc images.
You used the dnf install -y container-tools command to install the container-tools meta-package on the host. The meta-package contains all container tools, such as Podman, Buildah, and Skopeo for additional support and troubleshooting. These tools are required for obtaining assistance from Red Hat Support when you are building and installing the image.

Procedure

Create a Containerfile that includes the following instructions:

Example Containerfile for RHEL image mode

FROM registry.redhat.io/rhel{op-system-version-major}/rhel-bootc:9.x

ARG USHIFT_VER={product-version}
RUN dnf config-manager \
        --set-enabled rhocp-${USHIFT_VER}-for-rhel-{op-system-version-major}-$(uname -m)-rpms \
        --set-enabled fast-datapath-for-rhel-{op-system-version-major}-$(uname -m)-rpms
RUN dnf install -y firewalld microshift && \
    systemctl enable microshift && \
    dnf clean all

# Create a default 'redhat' user with the specified password.
# Add it to the 'wheel' group to allow for running sudo commands.
ARG USER_PASSWD
RUN if [ -z "${USER_PASSWD}" ] ; then \
        echo USER_PASSWD is a mandatory build argument && exit 1 ; \
    fi
RUN useradd -m -d /var/home/redhat -G wheel redhat && \
    echo "redhat:${USER_PASSWD}" | chpasswd

# Mandatory firewall configuration
RUN firewall-offline-cmd --zone=public --add-port=22/tcp && \
    firewall-offline-cmd --zone=trusted --add-source=10.42.0.0/16 && \
    firewall-offline-cmd --zone=trusted --add-source=169.254.169.1

# Create a systemd unit to recursively make the root filesystem subtree
# shared as required by OVN images
RUN cat > /etc/systemd/system/microshift-make-rshared.service <<'EOF'
[Unit]
Description=Make root filesystem shared
Before=microshift.service
ConditionVirtualization=container
[Service]
Type=oneshot
ExecStart=/usr/bin/mount --make-rshared /
[Install]
WantedBy=multi-user.target
EOF
RUN systemctl enable microshift-make-rshared.service

Podman uses the host subscription information and repositories inside the container when building the container image. If the rhocp and fast-datapath repositories are not available on the host, the build fails.

Set the PULL_SECRET environment variable:
```
$ PULL_SECRET=~/.pull-secret.json
```
Configure the USER_PASSWD environment variable:
```
$ USER_PASSWD=<redhat_user_password>
```
Replace <redhat_user_password> with your password.

Configure the IMAGE_NAME environment variable:

$ IMAGE_NAME=microshift-{product-version}-bootc

Create a local bootc image by running the following image build command:

$ sudo podman build --authfile "${PULL_SECRET}" -t "${IMAGE_NAME}" \
    --build-arg USER_PASSWD="${USER_PASSWD}" \
    -f Containerfile

How secrets are used during the image build:

The podman --authfile argument is required to pull the base rhel-bootc:9.x image from the registry.redhat.io registry.
The build USER_PASSWD argument is used to set a password for the redhat user.

Verification

Verify that the local {microshift-short} bootc image was created by running the following command:

$ sudo podman images "${IMAGE_NAME}"

Example output

REPOSITORY                       TAG         IMAGE ID      CREATED        SIZE
localhost/microshift-{product-version}-bootc  latest      193425283c00  2 minutes ago  2.31 GB

Publishing the bootc image to the remote registry

Publish your bootc image to the remote registry so that the image can be used for running the container on another host, or for when you want to install a new operating system with the bootc image layer.

Prerequisites

You are logged in to the RHEL 9.x host where the image was built using the user credentials that have sudo permissions.
You have a remote registry such as Red Hat Quay for storing and accessing bootc images.
You created the Containerfile and built the image.

Procedure

Set the REGISTRY_URL variable for the image by running the following command:
```
$ REGISTRY_URL=<quay.io>
```
Replace <quay.io> with the URL for your image registry.
Log in to your remote registry by running the following command:
```
$ sudo podman login "${REGISTRY_URL}"
```
Set the IMAGE_NAME variable for the image by running the following command:
```
$ IMAGE_NAME=<microshift-{product-version}-bootc>
```
Replace <microshift-{product-version}-bootc> with the name of the image you want to publish.
Set the REGISTRY_IMG variable for the image by running the following command:
```
$ REGISTRY_IMG=<myorg/mypath>/"${IMAGE_NAME}"
```
Replace <myorg/mypath> with your remote registry organization name and path.

Publish the image by running the following command:

$ sudo podman push localhost/"${IMAGE_NAME}" "${REGISTRY_URL}/${REGISTRY_IMG}"

Verification

Run the container using the image you pushed to your registry as described in the "Running the {microshift-short} bootc container" section.

About physically bound bootc image building

When a bootc image is fully self-contained, everything you need to run workloads is embedded with the bootc image, including {microshift-short} and application container images. The underlying mechanism is to pre-pull physically-bound images during image build and then make them available at runtime.

Because embedded images might change with each system update, you cannot pull the images directly to the default container storage. Additional image stores do not work in this case because of current implementation limits. These limits do not allow bootc image updates for those container images.

The manifest, layer tarballs, and signatures are exported as individual files into the directory. The dir transport type preserves the digest of the image, which is crucial for the original digest to reference the image.

Technical details to understand include the following items:

Each image goes into the same top-level directory, but a separate subdirectory.
Subdirectories are named after the image reference string SHA.
An image list file maps image references to their name SHA.
You must install the microshift-release-info RPM to get the image references required by {microshift-short}.
You must have image references for your workloads. Apply the same methods to workload image references that you use for {microshift-short} image references.
When you build the container, you must install the microshift-release-info RPM. The release-x86_64.json and release-aarch64.json files from this RPM reside in the /usr/share/microshift/release/ directory. These files contain image references required by {microshift-short}.

You must keep track of the name of the image. A tag, digest, or a mix of both can reference images. Choosing the best way to reference the images you need can impact the quality and robustness of workloads.

Embedding container images into a bootc image

You embed container images by adding instructions to an existing Containerfile to copy the images you want and list them in a file to keep track of the copied image names.

Then, you must copy images locally from the /usr/lib/containers/storage directory to the local container storage.

You cannot store images in the default or additional container storage directory when you build bootc images. For example, if you update the additional container store setting in /etc/containers/storage.conf to point to the /usr/lib/containers/storage directory, bootc image updates fail.

Prerequisites

You have root access to the host.
You installed Podman.
You installed skopeo.
You have workload image references.
You have a Containerfile for building {microshift-short} images.

Procedure

Add the pull secret to the container build procedure to ensure that images can be pulled by running the following command:
```
$ podman build --secret id=pullsecret,src=/<path/to/pull/secret>.json
```
Specify the path to your pull secret in <path/to/pull/secret>.

Add the instructions to physically embed the image at build time by adding the following to your Containerfile:

ENV IMAGE_STORAGE_DIR=/usr/lib/containers/storage
ENV IMAGE_LIST_FILE=${IMAGE_STORAGE_DIR}/image-list.txt

RUN dnf install -y microshift-release-info
RUN --mount=type=secret,id=pullsecret,dst=/run/secrets/pull-secret.json \
    images="$(jq -r ".images[]" /usr/share/microshift/release/release-"$(uname -m)".json)" ; \
    mkdir -p "${IMAGE_STORAGE_DIR}" ; \
    for img in ${images} ; do \
        sha="$(echo "${img}" | sha256sum | awk '{print $1}')" ; \
        skopeo copy --all --preserve-digests \
            --authfile /run/secrets/pull-secret.json \
            "docker://${img}" "dir:$IMAGE_STORAGE_DIR/${sha}" ; \
        echo "${img},${sha}" >> "${IMAGE_LIST_FILE}" ; \
    done

When run, the Containerfile extracts the list of {microshift-short} container image dependencies from the microshift-release-info RPM and pulls them into a custom /usr/lib/containers/storage directory. The resulting image list file is saved at /usr/lib/containers/storage/image-list.txt.

Next, you must copy container images from the custom directory to the main container storage directory so that they are available to {microshift-short}. Add a script and a systemd service to your Containerfile to copy the embedded images from the /usr/lib/containers/storage directory to the local container storage. Copying happens at runtime before each {microshift-short} start. Use the following example:

RUN cat > /usr/bin/microshift-copy-images <<EOF
#!/bin/bash
set -eux -o pipefail
while IFS="," read -r img sha ; do
    skopeo copy --preserve-digests \
        "dir:${IMAGE_STORAGE_DIR}/\${sha}" \
        "containers-storage:\${img}"
done < "${IMAGE_LIST_FILE}"
EOF

RUN chmod 755 /usr/bin/microshift-copy-images && \
    mkdir -p /usr/lib/systemd/system/microshift.service.d

RUN cat > /usr/lib/systemd/system/microshift.service.d/microshift-copy-images.conf <<EOF
[Service]
ExecStartPre=/usr/bin/microshift-copy-images
EOF

Next steps

Build the image.
Test and deploy per your use case.

Creating the Kickstart file

You must create the Kickstart file to use during installation.

Prerequisites

You have root-user access.
You are logged in to the physical hypervisor host.

Procedure

Set the AUTH_CONFIG environment variable to reference the secret file in the kickstart.ks file to authenticate private container registry access by running the following command:
```
$ AUTH_CONFIG=~/.quay-auth.json
```
Set the PULL_SECRET environment variable to reference the secret files in the kickstart.ks file to authenticate the {OCP} registry access by running the following command:
```
$ PULL_SECRET=~/.pull-secret.json
```
Set the IMAGE_REF environment variable to reference the image mode for your container image to use during installation by running the following command:
```
$ IMAGE_REF="quay.io/<myorg>/<mypath>/microshift-{product-version}-bootc"
```
Replace <myorg/<mypath> with your remote registry organization name and path.

Create the kickstart.ks file to use during installation by running the following script:

$ cat > kickstart.ks <<EOFKS
lang en_US.UTF-8
keyboard us
timezone UTC
text
reboot

# Partition the disk with hardware-specific boot and swap partitions, adding an
# LVM volume that contains a 10GB+ system root. The remainder of the volume will
# be used by the CSI driver for storing data.
zerombr
clearpart --all --initlabel
# Create boot and swap partitions as required by the current hardware platform
reqpart --add-boot
# Add an LVM volume group and allocate a system root logical volume
part pv.01 --grow
volgroup rhel pv.01
logvol / --vgname=rhel --fstype=xfs --size=10240 --name=root

# Lock root user account
rootpw --lock

# Configure network to use DHCP and activate on boot
network --bootproto=dhcp --device=link --activate --onboot=on

%pre-install --log=/dev/console --erroronfail

# Create a 'bootc' image registry authentication file
mkdir -p /etc/ostree
cat > /etc/ostree/auth.json <<'EOF'
$(cat "${AUTH_CONFIG}")
EOF

%end

# Pull a 'bootc' image from a remote registry
ostreecontainer --url "${IMAGE_REF}"

%post --log=/dev/console --erroronfail

# Create an OpenShift pull secret file
cat > /etc/crio/openshift-pull-secret <<'EOF'
$(cat "${PULL_SECRET}")
EOF
chmod 600 /etc/crio/openshift-pull-secret

%end
EOFKS

Creating a virtual machine

You can create a virtual machine by using the Red Hat Enterprise Linux (RHEL) boot ISO image.

Prerequisites

You created the Kickstart file.
You installed the OpenShift CLI (oc).
You have redhat credentials.

Procedure

Download the Red Hat Enterprise Linux (RHEL) boot ISO image from the Download Red Hat Enterprise Linux.
Copy the downloaded file to the /var/lib/libvirt/images directory.
Configure the VMNAME environment variable with your value by running the following command:
```
$ VMNAME=microshift-{product-version}-bootc
```
Configure the NETNAME environment variable with your value by running the following command:
```
$ NETNAME=default
```

Create a RHEL virtual machine with 2 cores, 2GB of RAM and 20GB of storage by running the following command:

$ sudo virt-install \
    --name ${VMNAME} \
    --vcpus 2 \
    --memory 2048 \
    --disk path=/var/lib/libvirt/images/${VMNAME}.qcow2,size=20 \
    --network network=${NETNAME},model=virtio \
    --events on_reboot=restart \
    --location /var/lib/libvirt/images/rhel-9.x-$(uname -m)-boot.iso \
    --initrd-inject kickstart.ks \
    --extra-args "inst.ks=file://kickstart.ks" \
    --wait

The sudo virt-install command uses the Kickstart file to pull a bootc image from the remote registry and install the RHEL operating system.

Verification

Verify that all of the {microshift-short} pods are running without error by entering the following command:

$ watch sudo oc get pods -A \
    --kubeconfig /var/lib/microshift/resources/kubeadmin/kubeconfig

Example output

NAMESPACE                  NAME                                       READY   STATUS    RESTARTS      AGE
kube-system                csi-snapshot-controller-7cfb9df49c-kc9dx   1/1     Running   0             31s
openshift-dns              dns-default-rpnlt                          2/2     Running   0             14s
openshift-dns              node-resolver-rxvdk                        1/1     Running   0             31s
openshift-ingress          router-default-69cd7b5545-7zcw7            1/1     Running   0             29s
openshift-ovn-kubernetes   ovnkube-master-c7hlh                       4/4     Running   1 (16s ago)   31s
openshift-ovn-kubernetes   ovnkube-node-mkpht                         1/1     Running   1 (17s ago)   31s
openshift-service-ca       service-ca-5d5d96459d-5pd5s                1/1     Running   0             28s
openshift-storage          topolvm-controller-677cbfcdb9-28dqr        5/5     Running   0             31s
openshift-storage          topolvm-node-6fzbl                         3/3     Running   0             14s

Creating a Kickstart file for installing {microshift-short} on image mode for RHEL

You can use the Kickstart file provided with {microshift-short} for an image mode for RHEL installation.

Prerequisites

You set up the {microshift-short} Kickstart file.
You have the information needed to set required and optional variables.

Procedure

Set the required BOOTC_IMAGE_URL variable value to point to an image used in the RHEL Kickstart file installation process by running the following command:
```
$ export BOOTC_IMAGE_URL=<myregistry>/<myorg>/<mypath>/microshift-image:tag
```
Replace <myregistry>, <myorg>, and <mypath> with your information.
- The BOOTC_IMAGE_URL variable contains a reference to the image that the ostreecontainer command installs. You can specify a z-stream release other than the latest by using the tag if required.
Optional. Add variables and values for registry authentication and configuration by using the following commands:
- Set the AUTH_CONFIG variable to authenticate access to the BOOTC_IMAGE_URL image by running the following command:
  $ export AUTH_CONFIG="$(cat ~/.quay-auth.json)"
  In this example, see the containers-auth.json(5) manual page for more information about this file format.
- Set the REGISTRY_CONFIG variable to configure access to the registry containing the BOOTC_IMAGE_URL image by running the following command:
  $ export REGISTRY_CONFIG="$(cat ~/.quay-config.conf)"
  In this example, see the containers-registries.conf(5) manual page for more information about this file format.

Create the kickstart.ks file to be used during the installation by running the following command:

envsubst < \
    /usr/share/microshift/kickstart/kickstart-bootc.ks.template > \
    "${HOME}/kickstart.ks"

Preparing for image building

Use the image builder tool to compose customized {op-system-ostree-first} images optimized for edge deployments.

This procedure is for use with RHEL 9 images only. RHEL 10 images require bootc image building tools. For more information, see the following link:

Composing, installing, and managing RHEL for Edge images

You can run a {microshift-short} node with your applications on a {op-system-ostree} virtual machine for development and testing first, then use your whole solution in edge production environments.

Use the following RHEL documentation to understand the full details of using {op-system-ostree}:

Introduction to RHEL for Edge images
To build an {op-system-ostree-first} 9.x image for a given CPU architecture, you need a RHEL 9.x build host of the same CPU architecture that meets the image builder system requirements. See the following link for more information:
- Image builder system requirements
To install image builder and the composer-cli tool, use the following instructions:
- Installing image builder

Enabling extended support repositories for image building

If you have an extended support (EUS) release of {microshift-short} or Red Hat Enterprise Linux (RHEL), you must enable the RHEL EUS repositories for image builder to use. If you do not have an EUS version, you can skip these steps.

Prerequisites

You have either an EUS version of {microshift-short} or RHEL, or you are updating to one.
You have root-user access to your build host.

You have reviewed the following link:

{op-system-bundle} release compatibility matrix

Keeping component versions in a supported configuration of {op-system-bundle} can require updating {microshift-short} and RHEL at the same time. Ensure that your version of RHEL is compatible with the version of {microshift-short} you are updating to, especially if you are updating {microshift-short} across two minor versions. Otherwise, you can create an unsupported configuration, break your node, or both. For more information, see the following link:

Red Hat Device Edge release compatibility matrix

Procedure

Create the /etc/osbuild-composer/repositories directory by running the following command:
```
$ sudo mkdir -p /etc/osbuild-composer/repositories
```
Copy the /usr/share/osbuild-composer/repositories/rhel-9.x.json file into the /etc/osbuild-composer/repositories directory by running the following command:
```
$ sudo cp /usr/share/osbuild-composer/repositories/rhel-9.x.json /etc/osbuild-composer/repositories/rhel-9.x.json
```
Update the baseos source by modifying the /etc/osbuild-composer/repositories/rhel-9.x.json file with the following values:
```
# ...
"baseurl": "https://cdn.redhat.com/content/eus/rhel{op-system-version-major}/9.x//baseos/os",
# ...
```
You can replace {op-system-version-major} with the major RHEL version you are using if different from the value in this example, and replace 9.x with the <major.minor> version. Be certain that the RHEL version you choose is compatible with the {microshift-short} version you are using.
Optional: Apply the baseos update by running the following command:
```
$ sudo sed -i "s,dist/rhel{op-system-version-major}/9.x/$(uname -m)/baseos/,eus/rhel{op-system-version-major}/9.x/$(uname -m)/baseos/,g" \
/etc/osbuild-composer/repositories/rhel-9.x.json
```
You can replace {op-system-version-major} with the major RHEL version you are using if different from the value in this example, and replace 9.x with the <major.minor> version. Be certain that the RHEL version you choose is compatible with the {microshift-short} version you are using.
Update the appstream source by modifying the /etc/osbuild-composer/repositories/rhel-<major.minor>.json file with the following values:
```
# ...
"baseurl": "https://cdn.redhat.com/content/eus/rhel{op-system-version-major}/9.x//appstream/os",
# ...
```
You can replace {op-system-version-major} with the major RHEL version you are using if different from the value in this example, and replace 9.x with the <major.minor> version. Be certain that the RHEL version you choose is compatible with the {microshift-short} version you are using.
Optional. Apply the appstream update by running the following command:
```
$ sudo sed -i "s,dist/rhel{op-system-version-major}/9.x/$(uname -m)/appstream/,eus/rhel{op-system-version-major}/9.x/$(uname -m)/appstream/,g" \
/etc/osbuild-composer/repositories/rhel-9.x.json
```
You can replace {op-system-version-major} with the major RHEL version you are using if different from the value in this example, and replace 9.x with the <major.minor> version. Be certain that the RHEL version you choose is compatible with the {microshift-short} version you are using.

Verification

Verify the baseos source by running the following command:

$ sudo composer-cli sources info baseos | grep 'url ='

Example output

url = "https://cdn.redhat.com/content/eus/rhel{op-system-version-major}/9.x/x86_64/baseos/os"

Verify the appstream source by running the following command:

$ sudo composer-cli sources info appstream | grep 'url ='

Example output

url = "https://cdn.redhat.com/content/eus/rhel{op-system-version-major}/9.x/x86_64/appstream/os"

Adding {microshift-short} repositories to image builder

Add the {microshift-short} repositories to image builder on your build host.

This procedure is for use with RHEL 9 images only. RHEL 10 images require bootc image building tools. For more information, see the following link:

Composing, installing, and managing RHEL for Edge images

Prerequisites

Your build host meets the image builder system requirements.
You have installed and set up image builder and the composer-cli tool.
You have root-user access to your build host.

Procedure

Create an image builder configuration file for adding the {rpm-repo-version} RPM repository source required to pull {microshift-short} RPMs by running the following command:

cat > {rpm-repo-version}.toml <<EOF
id = "{rpm-repo-version}"
name = "Red Hat OpenShift Container Platform {ocp-version} for RHEL {op-system-version-major}"
type = "yum-baseurl"
url = "https://cdn.redhat.com/content/dist/layered/rhel9/$(uname -m)/rhocp/{ocp-version}/os"
check_gpg = true
check_ssl = true
system = false
rhsm = true
EOF

Create an image builder configuration file for adding the fast-datapath RPM repository by running the following command:

cat > fast-datapath.toml <<EOF
id = "fast-datapath"
name = "Fast Datapath for RHEL 9"
type = "yum-baseurl"
url = "https://cdn.redhat.com/content/dist/layered/rhel9/$(uname -m)/fast-datapath/os"
check_gpg = true
check_ssl = true
system = false
rhsm = true
EOF

Add the sources to the image builder by running the following commands:

$ sudo composer-cli sources add {rpm-repo-version}.toml

$ sudo composer-cli sources add fast-datapath.toml

Verification

Confirm that the sources were added properly by running the following command:
```
$ sudo composer-cli sources list
```
Example output
```
appstream
baseos
fast-datapath
{rpm-repo-version}
```

Next steps

Create the blueprint. For more information, see the following links:

Adding the {microshift-short} service to a blueprint

Adding the {microshift-short} RPM package to an image builder blueprint enables the build of a {op-system-ostree} image with {microshift-short} embedded.

Procedure

Use the blueprint installed in the /usr/share/microshift/blueprint directory that is specific to your platform architecture. See the following example snippet for an explanation of the blueprint sections:
Generated image builder blueprint example snippet
```
name = "microshift_blueprint"
description = "MicroShift {ocp-version}.1 on x86_64 platform"
version = "0.0.1"
modules = []
groups = []

[[packages]]
name = "microshift"
version = "{ocp-version}.1"
...
...

[customizations.services]
enabled = ["microshift"]

[customizations.firewall]
ports = ["ssh"]
...
...

[[containers]]
source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41e79c17e8b41f1b0a5a32c3e2dd7cd15b8274554d3f1ba12b2598a347475f4"

[[containers]]
source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbc65f1fba7d92b36cf7514cd130fe83a9bd211005ddb23a8dc479e0eea645fd"
...
…
EOF
```
- [[packages]] name = "microshift": references for all non-optional {microshift-short} RPM packages using the same version compatible with the microshift-release-info RPM.
- [customizations.services] enabled = ["microshift"]: references for automatically enabling {microshift-short} on system startup and applying default networking settings.
- [[containers]] source = "quay.io/openshift-release-dev/…: references for all non-optional {microshift-short} container images necessary for an offline deployment. The SHA depends on the release you are using.
Add the blueprint to the image builder by running the following command:
```
$ sudo composer-cli blueprints push microshift_blueprint.toml
```

Verification

Verify the image builder configuration listing only {microshift-short} packages by running the following command:

$ sudo composer-cli blueprints depsolve microshift_blueprint | grep microshift

Example output

blueprint: microshift_blueprint v0.0.1
    microshift-release-info-{ocp-version}.1-202511250827.p0.g4105d3b.assembly.{ocp-version}.1.el9.noarch
    microshift-{ocp-version}.1-202511250827.p0.g4105d3b.assembly.{ocp-version}.1.el9.x86_64

Optional: Verify the image builder configuration that lists all of the components to be installed by running the following command:
```
$ sudo composer-cli blueprints depsolve microshift_blueprint
```

Adding other packages to a blueprint

Add the references for optional RPM packages to your ostree blueprint to enable them.

Prerequisites

You created an image builder blueprint file.

Procedure

Edit your ostree blueprint by running the following command:
```
$ vi <microshift_blueprint.toml>
```
Replace <microshift_blueprint.toml> with the name of the blueprint file used for the {microshift-short} service.
Add the following example text to your blueprint:
```
[[packages]]
name = "<microshift_additional_package_name>"
version = "*"
```
- [[packages]] name = Include one stanza for each additional service that you want to add. For example, replace <microshift_additional_package_name> in with the name the RPM for the service you want to include such as microshift-olm. Add another stanza as needed.

Next steps

Add custom certificate authorities to the blueprint as needed. For more information, see the following links:
After you finish adding to your blueprint, you can apply the manifests to an active node by building a new {op-system-ostree} system and deploying it on the client:
- Create the ISO.
- Add the blueprint and build the ISO.
- Download the ISO and prepare it for use.
- Do any provisioning that is needed.

Certificate authority bundle configuration

{microshift-short} uses the host trust bundle when clients evaluate server certificates.

You can also use a customized security certificate chain to improve the compatibility of your endpoint certificates with clients specific to your deployments. To do this, you can add a certificate authority (CA) bundle with root and intermediate certificates to the {op-system-ostree-first} system-wide truststore.

Adding a certificate authority bundle to a blueprint

You can include additional certificate authorities (CAs) to be trusted by the operating system when pulling images from an image registry. To add the additional CAs to the {op-system-ostree-first} rpm-ostree image, configure them in the blueprint that you use to create the image.

This procedure requires you to configure the CA bundle customizations in the blueprint, and then add steps to your Kickstart file to enable the bundle. In the following steps, data is the key, and <value> represents the PEM-encoded certificate.

Prerequisites

You have root user access to your build host.
Your build host meets the image builder system requirements.
You have installed and set up image builder and the composer-cli tool.

Procedure

Add the following custom values to your blueprint to add a directory.
1. Add instructions to your blueprint on the host where the image is built to create the directory, for example, /etc/pki/ca-trust/source/anchors/ for your certificate bundles.
  [[customizations.directories]] path = "/etc/pki/ca-trust/source/anchors"
2. After the image has booted, create the certificate bundles, for example, /etc/pki/ca-trust/source/anchors/cert1.pem:
  [[customizations.files]] path = "/etc/pki/ca-trust/source/anchors/cert1.pem" data = "<value>"

To enable the certificate bundle in the system-wide truststore configuration, use the update-ca-trust command on the host where the image you are using has booted, for example:

$ sudo update-ca-trust

The update-ca-trust command might be included in the %post section of a Kickstart file used for {microshift-short} host installation so that all the necessary certificate trust is enabled on the first boot. You must configure the CA bundle customizations in the blueprint before adding steps to your Kickstart file to enable the bundle.

%post
# Update certificate trust storage in case new certificates were
# installed at /etc/pki/ca-trust/source/anchors directory
update-ca-trust
%end

Creating the {op-system-ostree} image with image builder

The {op-system-ostree} Installer image pulls the commit from the running container and creates an installable boot ISO with a Kickstart file configured to use the embedded rpm-ostree commit.

Prerequisites

Your build host meets the image builder system requirements.
You installed and set up image builder and the composer-cli tool.
You root-user access to your build host.
You installed the podman tool.

Procedure

Start an ostree container image build by running the following command:
```
$ BUILDID=$(sudo composer-cli compose start-ostree --ref "rhel/{op-system-version-major}/$(uname -m)/edge" <microshift_blueprint> edge-container | awk '/^Compose/ {print $2}')
```
Replace <microshift_blueprint> with the name of your blueprint.

This command also returns the identification (ID) of the build for monitoring.

You can check the status of the build periodically by running the following command:

$ sudo composer-cli compose status

Example output of a running build

ID                                     Status     Time                     Blueprint                 Version   Type               Size
cc3377ec-4643-4483-b0e7-6b0ad0ae6332   RUNNING    Wed Jun 7 12:26:23 2023  microshift_blueprint      0.0.1     edge-container

Example output of a completed build

ID                                     Status     Time                      Blueprint              Version   Type               Size
cc3377ec-4643-4483-b0e7-6b0ad0ae6332   FINISHED   Wed Jun 7 12:32:37 2023   microshift_blueprint   0.0.1     edge-container

You can use the watch command to monitor your build if you are familiar with how to start and stop it.

Download the container image using the ID and get the image ready for use by running the following command:
```
$ sudo composer-cli compose image ${BUILDID}
```
Change the ownership of the downloaded container image to the current user by running the following command:
```
$ sudo chown $(whoami). ${BUILDID}-container.tar
```
Add read permissions for the current user to the image by running the following command:
```
$ sudo chmod a+r ${BUILDID}-container.tar
```
Bootstrap a server on port 8085 for the ostree container image to be consumed by the ISO build by completing the following steps:
1. Get the IMAGEID variable result by running the following command:
  $ IMAGEID=$(cat < "./${BUILDID}-container.tar" | sudo podman load | grep -o -P '(?<=sha256[@:])[a-z0-9]*')
2. Use the IMAGEID variable result to run the Podman command step by running the following command:
  $ sudo podman run -d --name=minimal-microshift-server -p 8085:8080 ${IMAGEID}
  This command also returns the ID of the container saved in the IMAGEID variable for monitoring.

Generate the installation program blueprint file by running the following command:

cat > microshift-installer.toml <<EOF
name = "microshift-installer"

description = ""
version = "0.0.0"
modules = []
groups = []
packages = []
EOF

Add the blueprint to image builder and build the ISO

You must add the blueprint to an image builder to build the ISO.

Procedure

Add the blueprint to the image builder by running the following command:
```
$ sudo composer-cli blueprints push microshift-installer.toml
```

Start the ostree ISO build by running the following command:

$ BUILDID=$(sudo composer-cli compose start-ostree --url http://localhost:8085/repo/ --ref "rhel/{op-system-version-major}/$(uname -m)/edge" microshift-installer edge-installer | awk '{print $2}')

This command also returns the identification (ID) of the build for monitoring.

You can check the status of the build periodically by running the following command:

$ sudo composer-cli compose status

Example output for a running build

ID                                     Status     Time                      Blueprint              Version   Type               Size
c793c24f-ca2c-4c79-b5b7-ba36f5078e8d   RUNNING    Wed Jun 7 13:22:20 2023   microshift-installer   0.0.0     edge-installer

Example output for a completed build

ID                                     Status     Time                      Blueprint              Version   Type               Size
c793c24f-ca2c-4c79-b5b7-ba36f5078e8d   FINISHED   Wed Jun 7 13:34:49 2023   microshift-installer   0.0.0     edge-installer

Download the ISO and prepare it for use

After creating the ISO, you must download it and prepare it for use.

Procedure

Download the ISO using the ID by running the following command:
```
$ sudo composer-cli compose image ${BUILDID}
```
Change the ownership of the downloaded container image to the current user by running the following command:
```
$ sudo chown $(whoami). ${BUILDID}-installer.iso
```
Add read permissions for the current user to the image by running the following command:
```
$ sudo chmod a+r ${BUILDID}-installer.iso
```

Next steps

Provision a virtual machine with a Kickstart file.

Embedding a Kickstart file in an ISO

You can use the Kickstart file provided with {microshift-short}, or you can update an existing {op-system-ostree} Installer (ISO) Kickstart file.

When ready, embed the Kickstart file into the ISO. Your Kickstart file must include detailed instructions about how to create a user and how to fetch and deploy the {op-system-ostree} image.

Prerequisites

You created a {op-system-ostree} Installer (ISO) image containing your {op-system-ostree} commit with {microshift-short}.
You have an existing Kickstart file ready for updating. You can use the microshift-starter.ks Kickstart file provided with the {microshift-short} RPMs.

Procedure

In the main section of the Kickstart file, update the setup of the filesystem such that it contains an LVM volume group called rhel with at least 10GB system root. Leave free space for the LVMS CSI driver to use for storing the data for your workloads.

Example Kickstart file snippet for configuring the filesystem

# Partition disk such that it contains an LVM volume group called `rhel` with a
# 10GB+ system root but leaving free space for the LVMS CSI driver for storing data.
#
# For example, a 20GB disk would be partitioned in the following way:
#
# NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
# sda             8:0    0  20G  0 disk
# ├─sda1          8:1    0 200M  0 part /boot/efi
# ├─sda1          8:1    0 800M  0 part /boot
# └─sda2          8:2    0  19G  0 part
#  └─rhel-root  253:0    0  10G  0 lvm  /sysroot
#
ostreesetup --nogpg --osname=rhel --remote=edge \
--url=file:///run/install/repo/ostree/repo --ref=rhel/<RHEL VERSION NUMBER>/x86_64/edge
zerombr
clearpart --all --initlabel
part /boot/efi --fstype=efi --size=200
part /boot --fstype=xfs --asprimary --size=800
# Uncomment this line to add a SWAP partition of the recommended size
#part swap --fstype=swap --recommended
part pv.01 --grow
volgroup rhel pv.01
logvol / --vgname=rhel --fstype=xfs --size=10000 --name=root
# To add users, use a line such as the following
user --name=<YOUR_USER_NAME> \
--password=<YOUR_HASHED_PASSWORD> \
--iscrypted --groups=<YOUR_USER_GROUPS>

In the %post section of the Kickstart file, add your pull secret and the mandatory firewall rules.

Example Kickstart file snippet for adding the pull secret and firewall rules

%post --log=/var/log/anaconda/post-install.log --erroronfail

# Add the pull secret to CRI-O and set root user-only read/write permissions
cat > /etc/crio/openshift-pull-secret << EOF
YOUR_OPENSHIFT_PULL_SECRET_HERE
EOF
chmod 600 /etc/crio/openshift-pull-secret

# Configure the firewall with the mandatory rules for MicroShift
firewall-offline-cmd --zone=trusted --add-source=10.42.0.0/16
firewall-offline-cmd --zone=trusted --add-source=169.254.169.1

%end

Install the mkksiso tool by running the following command:
```
$ sudo yum install -y lorax
```

Update the ISO with your new Kickstart file by running the following command:

$ sudo mkksiso <your_kickstart>.ks <your_installer>.iso <updated_installer>.iso

Additional resources

About offline deployments with {op-system-ostree}

Embedding {microshift-short} containers in an rpm-ostree commit means that you can run a node in disconnected or offline environments.

You can embed {product-title} containers in a {op-system-ostree-first} image so that container engines do not need to pull images over a network from a container registry. Workloads can start immediately without network connectivity.

Embedding {microshift-short} containers for offline deployments

You can use image builder to create {op-system-ostree} images with embedded {microshift-short} container images. To embed container images, you must add the image references to your image builder blueprint file.

This procedure is for use with RHEL 9 images only. RHEL 10 images require bootc image building tools. For more information, see the following link:

Composing, installing, and managing RHEL for Edge images

Prerequisites

You have root-user access to your build host.
Your build host meets the image builder system requirements.
You installed and set up image builder and the composer-cli tool.
You created a {op-system-ostree} image blueprint.
You installed jq.

Procedure

Get the exact list of container image references used by the {microshift-short} version you are deploying. You can either install the microshift-release-info RPM package by following step 2 or download and unpack the RPM by following step 3.
To install the microshift-release-info RPM package:
1. Install the microshift-release-info RPM package by running the following command:
  $ sudo dnf install -y microshift-release-info-<release_version>
  Replace <release_version> with the numerical value of the release you are deploying, using the entire version number, such as 4.22.0.
2. List the contents of the /usr/share/microshift/release directory to verify the presence of the release information files by running the following command:
  $ sudo ls /usr/share/microshift/release
  Example output
  release-x86_64.json release-aarch64.json
  If you installed the microshift-release-info RPM, proceed to step 4.
If you did not complete step 2, download and unpack the microshift-release-info RPM without installing it:
1. Download the RPM package by running the following command:
  $ sudo dnf download microshift-release-info-<release_version>
  Replace <release_version> with the numerical value of the release you are deploying, using the entire version number, such as 4.22.0.
  Example RPM output
  microshift-release-info-4.22.0.-202605191402.p0.g4f61957.assembly.rc.4.el9.noarch.rpm
2. Unpack the RPM package without installing it by running the following command:
  $ rpm2cpio <my_microshift_release_info> | cpio -idmv ./usr/share/microshift/release/release-aarch64.json ./usr/share/microshift/release/release-x86_64.json
  Replace <my_microshift_release_info> with the name of the RPM package from the previous step.
Define the location of your JSON file, which contains the container reference information, by running the following command:
```
$ RELEASE_FILE=</path/to/your/release-$(uname -m).json>
```
Replace </path/to/your/release-$(uname -m).json> with the full path to your JSON file. Be sure to use the file needed for your architecture.
Define the location of your TOML file, which contains instructions for building the image, by running the following command:
```
$ BLUEPRINT_FILE=</path/to/your/blueprint.toml>
```
Replace </path/to/your/blueprint.toml> with the full path to your TOML file.

Generate and then embed the container image references in your blueprint TOML file by running the following command:

$  jq -r '.images | .[] | ("[[containers]]\nsource = \"" + . + "\"\n")' "${RELEASE_FILE}" >> "${BLUEPRINT_FILE}"

Example resulting TOML fragment showing container references

[[containers]]
source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82cfef91557f9a70cff5a90accba45841a37524e9b93f98a97b20f6b2b69e5db"

[[containers]]
source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82cfef91557f9a70cff5a90accba45841a37524e9b93f98a97b20f6b2b69e5db"

You can manually embed any container image by adding it to an image builder blueprint file using the following example:
Example section for manually embedding container image to a blueprint
```
[[containers]]
source = "<my_image_pullspec_with_tag_or_digest>"
```
Replace <my_image_pullspec_with_tag_or_digest> with the exact reference to a container image used by the {microshift-short} version you are deploying.

Adding registry authentication to prepare for image building

After you have updated the blueprint, you must add authentication for the container registries to build the image with embedded {microshift-short} containers. To do this, update one of the systemd service files that are part of the image builder configuration.

Prerequisites

You have root-user access to your build host.
Your build host meets the image builder system requirements.
You have installed and set up image builder and the composer-cli tool.

Procedure

Create an /etc/osbuild-worker/osbuild-worker.toml directory and configuration file if they do not exist.
Add a pull secret for authenticating to the registry by setting the auth_file_path in the [containers] section of the /etc/osbuild-worker/osbuild-worker.toml configuration file:
```
[containers]
auth_file_path = "/etc/osbuild-worker/pull-secret.json"
```
Restart the host to apply configuration changes.

Build and use the rpm-ostree image for offline deployments

You can use image builder to create rpm-ostree system images with embedded {microshift-short} container images.

To embed container images, you must add the image references to your image builder blueprint. You can create the commit and ISO as needed for your use case.

Add the prerequisites listed here to the ones that are included in the procedures that follow.

Additional prerequisites for offline deployments

You have created and updated a {op-system-ostree} image blueprint for offline use. The following procedures use the example of a blueprint created with container images. You must use the updated blueprint you created in the "Embedding MicroShift containers for offline deployments" procedure.
You have updated the /etc/osbuild-worker/osbuild-worker.toml configuration file for offline use.

Replace minimal-microshift.toml in the following procedures with the name of the TOML you updated for offline use, <my_blueprint_name>.

Creating a Kickstart file for a {microshift-short} {op-system-ostree} installation

You can use the Kickstart file provided with {microshift-short} to provision a {op-system-ostree}-based virtual machine.

Prerequisites

You set up the {microshift-short} Kickstart file.
You have the information needed to set required and optional variables.

Procedure

Add the following required variables to create an {op-system-ostree} Kickstart file:
- The OSTREE_SERVER_URL variable contains an rpm-ostree server URL that is passed to the ostreesetup Kickstart command.
  $ export OSTREE_SERVER_URL="<http://my_ostree_server_url>"
  Replace http://my_ostree_server_url with your server URL.
- The OSTREE_COMMIT_REF variable contains an rpm-ostree commit reference that is installed from the server.
  $ export OSTREE_COMMIT_REF="<myostree_commit_reference>"
  Replace <myostree_commit_reference> with the rpm-ostree commit reference.
Optional. Add the following variable for server authentication:
- The AUTH_CONFIG contents are copied to /etc/ostree/auth.json at the pre-install stage to authenticate access to the OSTREE_SERVER_URL server. If no server authentication is required, skip this setting.
  $ export AUTH_CONFIG="$(cat ~/.ostree-auth.json)"
Run the following command to create the kickstart.ks file to be used during the installation:
```
envsubst < \
    /usr/share/microshift/kickstart/kickstart-ostree.ks.template > \
    "${HOME}/kickstart.ks"
```
The {microshift-short} version specified in the rpm-ostree commit is installed. To change the version of {microshift-short}, you must create a new commit.

Before installing {microshift-short} from an RPM package

Before installing {microshift-short} for memory configuration and FIPS mode, you must prepare the host.

Configuring volume groups

{microshift-short} uses the logical volume manager storage (LVMS) Container Storage Interface (CSI) plugin for providing storage to persistent volumes (PVs). LVMS relies on the Linux logical volume manager (LVM) to dynamically manage the backing logical volumes (LVs) for PVs. For this reason, your machine must have an LVM volume group (VG) with unused space in which LVMS can create the LVs for your workload’s PVs.

To configure a volume group (VG) that allows LVMS to create the LVs for your workload’s PVs, lower the Desired Size of your root volume during the installation of RHEL. Lowering the size of your root volume allows unallocated space on the disk for additional LVs created by LVMS at runtime.

Prepare for FIPS mode

If your use case requires running {microshift-short} containers in FIPS mode, you must install RHEL with FIPS enabled. After the worker machine is configured to run in FIPS mode, your {microshift-short} containers are automatically configured to also run in FIPS mode.

Because FIPS must be enabled before the operating system that your node uses starts for the first time, you cannot enable FIPS after you deploy a node.

Preparing to install {microshift-short} from an RPM package

When you are getting ready to install {microshift-short} RPMs, make sure you have enough storage capacity for the workload you want to run.

Prerequisites

The system requirements for installing {microshift-short} have been met.
You have root user access to your machine.
You have configured your LVM VG with the capacity needed for the PVs of your workload.

Procedure

In the graphical installer under Installation Destination in the Storage Configuration subsection, select Custom → Done to open the dialog for configuring partitions and volumes. The Manual Partitioning window is displayed.
Under New Red Hat Enterprise Linux {op-system-version-major}.x Installation, select Click here to create them automatically.
Select the root partition, /, reduce Desired Capacity so that the VG has sufficient capacity for your PVs, and then click Update Settings.
Complete your installation.

For more options on partition configuration, read the guide linked in the Additional information section for Configuring Manual Partitioning.
As a root user, verify the VG capacity available on your system by running the following command:
```
$ sudo vgs
```
Example output:
```
VG   #PV #LV #SN Attr   VSize    VFree
rhel   1   2   0 wz--n- 127.00g 54.94g
```

Installing {microshift-short} from an RPM package

Use the following procedure to install {microshift-short} from an RPM package.

Prerequisites

The system requirements for installing {microshift-short} have been met.
You completed the steps of preparing to install {microshift-short} from an RPM package.

Procedure

For all lifecycles, enable the repository for your release by running the following command:

$ sudo subscription-manager repos \
    --enable rhocp-{ocp-version}-for-rhel-{op-system-version-major}-$(uname -m)-rpms \
    --enable fast-datapath-for-rhel-{op-system-version-major}-$(uname -m)-rpms

For extended support (EUS) releases, also enable the EUS repositories by running the following command:

$ sudo subscription-manager repos \
    --enable rhel-{op-system-version-major}-for-$(uname -m)-appstream-eus-rpms \
    --enable rhel-{op-system-version-major}-for-$(uname -m)-baseos-eus-rpms

Avoid unintended future updates into an unsupported configuration by locking your operating system version with the following command:
```
$ sudo subscription-manager release --set=9.x
```
Install {microshift-short} by running the following command:
```
$ sudo dnf install -y microshift
```
Download your installation pull secret from the Red Hat Hybrid Cloud Console to a temporary folder, for example, $HOME/openshift-pull-secret. This pull secret allows you to authenticate with the container registries that serve the container images used by {product-title}.
To copy the pull secret to the /etc/crio folder of your RHEL machine, run the following command:
```
$ sudo cp $HOME/openshift-pull-secret /etc/crio/openshift-pull-secret
```
Make the root user the owner of the /etc/crio/openshift-pull-secret file by running the following command:
```
$ sudo chown root:root /etc/crio/openshift-pull-secret
```
Make the /etc/crio/openshift-pull-secret file readable and writeable by the root user only by running the following command:
```
$ sudo chmod 600 /etc/crio/openshift-pull-secret
```

If your RHEL machine has a firewall enabled, you must configure a few mandatory firewall rules. For firewalld, run the following commands:

$ sudo firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16

$ sudo firewall-cmd --permanent --zone=trusted --add-source=169.254.169.1

$ sudo firewall-cmd --reload

If the Volume Group (VG) that you have prepared for {microshift-short} used the default name rhel, no further configuration is necessary. If you have used a different name, or if you want to change more configuration settings, see the "Using the {microshift-short} configuration file" section.

FIPS mode with RHEL RPM-based installations

Using FIPS with {microshift-short} requires enabling the cryptographic module self-checks in your Red Hat Enterprise Linux (RHEL) installation. After the host operating system has been configured to start with the FIPS modules, {microshift-short} containers are automatically enabled to run in FIPS mode.

When RHEL is started in FIPS mode, {microshift-short} core components use the RHEL cryptographic libraries that have been submitted to NIST for FIPS 140-2/140-3 validation on only the x86_64 architectures.
You must enable FIPS mode when you install RHEL {op-system-version-major} on the machines that you plan to use as worker machines.

Because FIPS must be enabled before the operating system that your node uses starts for the first time, you cannot enable FIPS after you deploy a node.
{microshift-short} uses a FIPS-compatible Golang compiler.
FIPS is supported in the CRI-O container runtime.

Limitations

TLS implementation FIPS support is not complete.
The FIPS implementation does not offer a single function that both computes hash functions and validates the keys that are based on that hash. This limitation continues to be evaluated for improvement in future {microshift-short} releases.

Installing the {gitops} Argo CD manifests from an RPM package

You can use a lightweight version of Red Hat OpenShift GitOps with {microshift-short} to help manage your applications by installing the microshift-gitops RPM package.

The microshift-gitops RPM package includes the necessary manifests to run core Argo CD.

The Argo CD web console is not available on {microshift-short}. This process installs basic {gitops} functions.

Prerequisites

You installed {microshift-short} version 4.16 or later.
You configured 250MB RAM of additional storage.

Procedure

Enable the {gitops} repository with the subscription manager by running the following command:
```
$ sudo subscription-manager repos --enable=gitops-1.1-for-{rhel-major}-$(uname -m)-rpms
```
Install the {microshift-short} {gitops} package by running the following command:
```
$ sudo dnf install -y microshift-gitops
```
To deploy Argo CD pods, restart {microshift-short} by running the following command:
```
$ sudo systemctl restart microshift
```

Verification

You can verify that your pods are running properly by entering the following command:

$ oc get pods -n openshift-gitops

Example output

NAME                                  READY   STATUS    RESTARTS   AGE
argocd-application-controller-0       1/1     Running   0          4m11s
argocd-redis-56844446bc-dzmhf         1/1     Running   0          4m12s
argocd-repo-server-57b4f896cf-7qk8l   1/1     Running   0          4m12s

Installing the multiple networks plugin

You can install the {microshift-short} Multus Container Network Interface (CNI) plugin alongside a new {microshift-short} installation. If you want to attach additional networks to a pod for high-performance network configurations, install the microshift-multus RPM package.

The {microshift-short} Multus CNI plugin manifests are included in the {microshift-short} binary. To enable multiple networks, you can either set the value in the {microshift-short} config.yaml file to Enabled, or use the configuration snippet in the microshift-multus RPM. Uninstalling the {microshift-short} Multus CNI is not supported in either case.

Procedure

Install the Multus RPM package by running the following command:
```
$ sudo dnf install microshift-multus
```
If you create your custom resources (CRs) while you are completing your installation of {microshift-short}, you can avoid restarting the service to apply them.

Next steps

Continue with your new {microshift-short} installation, including any add-ons.
Create the custom resources (CRs) needed for your {microshift-short} Multus CNI plugin.
Configure other networking CNIs as needed.
After you have finished installing all of the RPMs that you want to include, start the {microshift-short} service. The {microshift-short} Multus CNI plugin is automatically deployed.

Installing the Operator Lifecycle Manager (OLM) from an RPM package

When you install {microshift-short}, the Operator Lifecycle Manager (OLM) package is not installed by default. You can install the OLM on your {microshift-short} instance by using an RPM package. OLM helps you install, update, and manage the lifecycle of Kubernetes native applications (Operators) and their associated services running in each {microshift-short} node.

Procedure

Install the OLM package by running the following command:
```
$ sudo dnf install microshift-olm
```
To apply the manifest from the package to an active node, run the following command:
```
$ sudo systemctl restart microshift
```

Installing and enabling {microshift-short} Observability

You can install {microshift-short} Observability at any time, including during the initial {microshift-short} installation. Observability collects and transmits system data for monitoring and analysis, such as performance and usage metrics and error reporting.

Procedure

Install the microshift-observability RPM by entering the following command:
```
$ sudo dnf install microshift-observability
```
Enable the microshift-observability system service by entering the following command:
```
$ sudo systemctl enable microshift-observability
```
Start the microshift-observability system service by entering the following command:
```
$ sudo systemctl start microshift-observability
```
Restart {microshift-short} after the initial installation.
```
$ sudo systemctl restart microshift-observability
```
The installation is successful if there is no output after you start the microshift-observability RPM.

Installing the Red Hat OpenShift AI RPM

To use AI models in {microshift-short} deployments, install the Red Hat OpenShift AI (RHOAI) RPM with a new {microshift-short} installation. You can also install the RPM on an existing {microshift-short} instance if you restart the system.

The microshift-ai-model-serving RPM contains manifests that deploy kserve, with the raw deployment mode enabled, and ServingRuntimes objects in the redhat-ods-applications namespace.

Red Hat OpenShift AI is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Prerequisites

The system requirements for installing {microshift-short} have been met.
You have root user access to your machine.
The OpenShift CLI (oc) is installed.
You configured your LVM VG with the capacity needed for the PVs of your workload.
You have the RAM and disk space required for your AI model.
You configured the required accelerators, hardware, operating system, and {microshift-short} to provide the resources your model needs.
Your AI model is ready to use.

Procedure

Install the {microshift-short} AI-model-serving RPM package by running the following command:
```
$ sudo dnf install microshift-ai-model-serving
```
As a root user, restart the {microshift-short} service by entering the following command:
```
$ sudo systemctl restart microshift
```

Optional: Install the release information package by running the following command:

$ sudo dnf install microshift-ai-model-serving-release-info

The microshift-ai-model-serving-release-info RPM contains a JSON file with image references useful for offline procedures or deploying a copy of a ServingRuntime custom resource (CR) to your namespace during a bootc image build.

Verification

Verify that the kserve pod is running in the redhat-ods-applications namespace by entering the following command:

$ oc get pods -n redhat-ods-applications

Example output

NAME                                        READY   STATUS    RESTARTS   AGE
kserve-controller-manager-7fc9fc688-kttmm   1/1     Running   0          1h

Next steps

Create a namespace for your AI model.
Package your model into an OCI image.
Configure a model-serving runtime.
Verify that your model is ready for inferencing.
Make requests against the model server.

Creating a Kickstart file for a {microshift-short} RPM installation

You can use the Kickstart file provided with {microshift-short} to provision an RPM-based virtual machine.

Prerequisites

You set up the {microshift-short} Kickstart file.
You know the Activation Keys and organization ID to activate your Red Hat subscription.
You have the information needed to set the BOOTC_IMAGE_URL, AUTH_CONFIG, and REGISTRY_CONFIG required variables.

The subscription must include access to the rhocp-4.x-for-rhel-{op-system-version-major}-$(uname -m)-rpms and fast-datapath-for-rhel-{op-system-version-major}-$(uname -m)-rpms RPM repositories.

Procedure

Add the following variables to create an RPM Kickstart file:
- The RHSM_ORG variable contains a Red Hat Subscription Manager organization ID for the subscription registration command in the Kickstart file.
  $ export RHSM_ORG="$(cat ~/.rhsm-activation-org)"
- The RHSM_KEY variable contains a Red Hat Subscription Manager activation key for the subscription registration command in the Kickstart file.
  $ export RHSM_KEY="$(cat ~/.rhsm-activation-key)"
- The MICROSHIFT_VER variable references the {microshift-short} version to install using the 4.y format. The latest available .z version of the 4.y version set in this variable is automatically installed.
  $ export MICROSHIFT_VER={product-version}
  In this example, the latest .z of the minor version that you set is installed.
Run the following command to create the kickstart.ks file to be used during the virtual machine installation:
```
envsubst < \
    /usr/share/microshift/kickstart/kickstart-rpm.ks.template > \
    "${HOME}/kickstart.ks"
```

How to access the {microshift-short} node

Access the {microshift-short} service by using the OpenShift CLI (oc).

You can access the node from either the same machine running the {microshift-short} service or from a remote location.
You can use this access to observe and administer workloads.
When using the following steps, choose the kubeconfig file that has the hostname or IP address you want to connect to and place it in the relevant directory.

Accessing the {microshift-short} node locally

Use the following procedure to access the {microshift-short} node locally by using a kubeconfig file.

Prerequisites

You installed the OpenShift CLI (oc).

Procedure

Optional: to create a ~/.kube/ folder if your Red Hat Enterprise Linux (RHEL) machine does not have one, run the following command:
```
$ mkdir -p ~/.kube/
```
Copy the generated local access kubeconfig file to the ~/.kube/ directory by running the following command:
```
$ sudo cat /var/lib/microshift/resources/kubeadmin/kubeconfig > ~/.kube/config
```
Update the permissions on your ~/.kube/config file by running the following command:
```
$ chmod go-r ~/.kube/config
```

Verification

Verify that {microshift-short} is running by entering the following command:

$ oc get pods -A

Example output

NAMESPACE                   NAME                                                     READY   STATUS   RESTARTS  AGE
default                     i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running  0		    46m
kube-system                 csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running  0		    51m
openshift-dns               dns-default-45jl7                                        2/2     Running  0		    50m
openshift-dns               node-resolver-7wmzf                                      1/1     Running  0		    51m
openshift-ingress           router-default-78b86fbf9d-qvj9s                          1/1     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-master-5rfhh                                     4/4     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-node-gcnt6                                       1/1     Running  0		    51m
openshift-service-ca        service-ca-bf5b7c9f8-pn6rk                               1/1     Running  0		    51m
openshift-storage           topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running  0		    51m
openshift-storage           topolvm-node-rht2m                                       3/3     Running  0		    50m

This example output shows a basic {microshift-short} installation. If you installed optional RPMs, the status of pods running those services is displayed in the output.

Opening the firewall for remote access to the {microshift-short} node

You must open the firewall before a workstation user can access the {microshift-short} node remotely.

For this procedure, user@microshift is the user on the {microshift-short} host machine and is responsible for setting up that machine so that it can be accessed by a remote user on a separate workstation.

Prerequisites

You installed the OpenShift CLI (oc).
Your account has cluster administration privileges.

Procedure

As user@microshift on the {microshift-short} host, open the firewall port for the Kubernetes API server (6443/tcp) by running the following command:
```
[user@microshift]$ sudo firewall-cmd --permanent --zone=public --add-port=6443/tcp && sudo firewall-cmd --reload
```

Verification

As user@microshift, verify that {microshift-short} is running by entering the following command:

$ oc get pods -A

Example output

NAMESPACE                   NAME                                                     READY   STATUS   RESTARTS  AGE
default                     i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running  0		    46m
kube-system                 csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running  0		    51m
openshift-dns               dns-default-45jl7                                        2/2     Running  0		    50m
openshift-dns               node-resolver-7wmzf                                      1/1     Running  0		    51m
openshift-ingress           router-default-78b86fbf9d-qvj9s                          1/1     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-master-5rfhh                                     4/4     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-node-gcnt6                                       1/1     Running  0		    51m
openshift-service-ca        service-ca-bf5b7c9f8-pn6rk                               1/1     Running  0		    51m
openshift-storage           topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running  0		    51m
openshift-storage           topolvm-node-rht2m                                       3/3     Running  0		    50m

This example output shows a basic {microshift-short} installation. If you installed optional RPMs, the status of pods running those services is displayed in the output.

Accessing the {microshift-short} node remotely

Access the {microshift-short} service from a remote location by using a kubeconfig file.

The user@workstation login is used to access the host machine remotely. The <user> value in the procedure is the name of the user that user@workstation logs in with to the {microshift-short} host.

Prerequisites

You installed the OpenShift CLI (oc).
The user@microshift has opened the firewall from the local host.
You generated additional kubeconfig files.

Procedure

As user@workstation, create a ~/.kube/ folder if your Red Hat Enterprise Linux (RHEL) machine does not have one by running the following command:
```
[user@workstation]$ mkdir -p ~/.kube/
```
As user@workstation, set a variable for the hostname of your {microshift-short} host by running the following command:
```
[user@workstation]$ MICROSHIFT_MACHINE=<microshift_hostname>
```
Replace the value, <{microshift-short}_hostname>, with the either the name or the IP address of the host running {microshift}.
As user@workstation, copy the generated kubeconfig file that has the hostname or IP address you want to connect to from the RHEL machine running {microshift-short} to your local machine by running the following command:
```
[user@workstation]$ ssh <user>@$MICROSHIFT_MACHINE "sudo cat /var/lib/microshift/resources/kubeadmin/$MICROSHIFT_MACHINE/kubeconfig" > ~/.kube/config #
```
Replace <user> with your SSH login credentials.
As user@workstation, update the permissions on your ~/.kube/config file by running the following command:
```
$ chmod go-r ~/.kube/config
```

Verification

As user@workstation, verify that {microshift-short} is running by entering the following command:

$ oc get pods -A

Example output

NAMESPACE                   NAME                                                     READY   STATUS   RESTARTS  AGE
default                     i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running  0		    46m
kube-system                 csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running  0		    51m
openshift-dns               dns-default-45jl7                                        2/2     Running  0		    50m
openshift-dns               node-resolver-7wmzf                                      1/1     Running  0		    51m
openshift-ingress           router-default-78b86fbf9d-qvj9s                          1/1     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-master-5rfhh                                     4/4     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-node-gcnt6                                       1/1     Running  0		    51m
openshift-service-ca        service-ca-bf5b7c9f8-pn6rk                               1/1     Running  0		    51m
openshift-storage           topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running  0		    51m
openshift-storage           topolvm-node-rht2m                                       3/3     Running  0		    50m

This example output shows a basic {microshift-short} installation. If you installed optional RPMs, the status of pods running those services is displayed in the output.

Uninstalling {microshift-short} from an RPM package

When you want to uninstall {microshift-short}, you must first clean up all data, pods, and configurations before removing the RPM packages.

Prerequisites

You are logged into {microshift-short} as an administrator with root-user access.
You have filed a support case.
You have root access to the {microshift-short} node.

Procedure

Clean all your data by running the following command:
```
$ sudo microshift-cleanup-data --all
```
When you run the script with the --all argument, you perform the following clean up actions:
- Stop and disable all {microshift-short} services
- Delete all {microshift-short} pods
- Delete all container image storage
- Reset network configuration
- Delete the /var/lib/microshift data directory
- Delete OVN-K networking configuration
Run the following command:
```
$ sudo dnf remove -y microshift*
```
= Administer

Red Hat build of MicroShift

Install

Mirror container images into an existing registry

Getting the mirror registry container image list

Configuring mirroring prerequisites

Example mirror registry pull secret entry

Downloading container images

Uploading container images to a mirror registry

Configuring hosts for mirror registry access

About image mode for Red Hat Enterprise Linux (RHEL)

Preparing for bootc image building

The image mode for RHEL with {microshift-short} workflow

Required resources

Workflow

Getting the published bootc image for {microshift-short}

Building the bootc image

Publishing the bootc image to the remote registry

About physically bound bootc image building

Embedding container images into a bootc image

Creating the Kickstart file

Creating a virtual machine

Creating a Kickstart file for installing {microshift-short} on image mode for RHEL

Preparing for image building

Enabling extended support repositories for image building

Adding {microshift-short} repositories to image builder

Adding the {microshift-short} service to a blueprint

Adding other packages to a blueprint

Certificate authority bundle configuration

Adding a certificate authority bundle to a blueprint

Creating the {op-system-ostree} image with image builder

Add the blueprint to image builder and build the ISO

Download the ISO and prepare it for use

Embedding a Kickstart file in an ISO

About offline deployments with {op-system-ostree}

Embedding {microshift-short} containers for offline deployments

Adding registry authentication to prepare for image building

Build and use the rpm-ostree image for offline deployments

Additional prerequisites for offline deployments

Creating a Kickstart file for a {microshift-short} {op-system-ostree} installation

Before installing {microshift-short} from an RPM package

Configuring volume groups

Prepare for FIPS mode

Preparing to install {microshift-short} from an RPM package

Installing {microshift-short} from an RPM package

FIPS mode with RHEL RPM-based installations

Limitations

Installing the {gitops} Argo CD manifests from an RPM package

Installing the multiple networks plugin

Installing the Operator Lifecycle Manager (OLM) from an RPM package

Installing and enabling {microshift-short} Observability

Installing the Red Hat OpenShift AI RPM

Creating a Kickstart file for a {microshift-short} RPM installation

How to access the {microshift-short} node

Accessing the {microshift-short} node locally

Opening the firewall for remote access to the {microshift-short} node

Accessing the {microshift-short} node remotely

Uninstalling {microshift-short} from an RPM package

API

Configure

Develop

Upgrade

Plan