Red Hat build of MicroShift

Upgrade

MicroShift updates on an RHEL for Edge system

Updating MicroShift on a Red Hat Enterprise Linux for Edge (RHEL for Edge) system requires building a new RHEL for Edge image containing the new version of MicroShift and any associated optional RPMs.

After you create the rpm-ostree image with MicroShift embedded, you can boot into that operating system image.

The procedures are the same for minor-version and patch updates. For example, use the same steps to upgrade from 4.20 to 4.21 or from 4.21.2 to 4.21.3. The following details apply:

Back up and system rollback are automatic with this update type.
You can use the following workflow to update applications running in the MicroShift node. Ensure compatibilities between the application and the adjacent versions of MicroShift and RHEL for Edge before starting an update.

Downgrades other than automatic rollbacks are not supported. The following procedure is for updates only.

The steps you use depends on how your existing deployment is set up. The following procedure outlines the general steps you can take, with links to the RHEL for Edge documentation. The RHEL for Edge documentation is your resource for specific details on building an updated operating system image.

Update MicroShift on RHEL for Edge

You can have automated backup and system rollback in case any part of the update fails by using the rpm-ostree update path for a new or existing RHEL for Edge deployment.

You can update MicroShift on an rpm-ostree system such as RHEL for Edge by building a new system image containing the new version of MicroShift.
The rpm-ostree image can be the same version or an updated version, but the versions of RHEL for Edge and MicroShift must be compatible.

The following features are available in the RHEL for Edge update path:

The system automatically rolls back to an earlier healthy system state if the update fails.
You do not need to reinstall applications.
You do not need to reinstall Operators.
You can update an application without updating MicroShift using this update type.
The image you build can contain other updates as needed.

To begin a MicroShift update by embedding the new version in a RHEL for Edge image, use the procedures in the following documentation:

Applying updates on a RHEL for Edge system

To understand more about greenboot, see the following documentation:

Apply updates on an RHEL for Edge system

To update MicroShift on Red Hat Enterprise Linux for Edge (RHEL for Edge), embed the new version of MicroShift on a new operating system image.

This procedure is for use with RHEL 9 images only. RHEL 10 images require bootc image building tools. For more information, see the following link:

Composing, installing, and managing RHEL for Edge images

You cannot downgrade MicroShift with this process. Downgrades other than automatic rollbacks are not supported.

Prerequisites

The system requirements for installing MicroShift have been met.
You have root user access to the host.
The version of MicroShift you have is compatible with the RHEL for Edge image you are preparing to use.

Procedure

Create an image builder configuration file for adding the rhocp-4.22 RPM repository source required to pull MicroShift RPMs by running the following command:

$ cat > rhocp-4.22.toml <<EOF
id = "rhocp-4.22"
name = "Red Hat OpenShift Container Platform 4.22 for RHEL 9"
type = "yum-baseurl"
url = "https://cdn.redhat.com/content/dist/layered/rhel9/$(uname -m)/rhocp/4.22/os"
check_gpg = true
check_ssl = true
system = false
rhsm = true
EOF

Add the update RPM source to the image builder by running the following command:
```
$ sudo composer-cli sources add rhocp-4.22.toml
```
Build a new image of RHEL for Edge that contains the new version of MicroShift. To determine the steps required, use the following documentation:
- Building a commit update
Update the host to use the new image of RHEL for Edge. To determine the steps required, use the following documentation:
- How RHEL for Edge image updates are deployed
Reboot the host to apply updates by running the following command:
```
$ sudo systemctl reboot
```

Update MicroShift on RHEL for Edge

You can have automated backup and system rollback in case any part of the update fails by using the rpm-ostree update path for a new or existing RHEL for Edge deployment.

You can update MicroShift on an rpm-ostree system such as RHEL for Edge by building a new system image containing the new version of MicroShift.
The rpm-ostree image can be the same version or an updated version, but the versions of RHEL for Edge and MicroShift must be compatible.

The following features are available in the RHEL for Edge update path:

The system automatically rolls back to an earlier healthy system state if the update fails.
You do not need to reinstall applications.
You do not need to reinstall Operators.
You can update an application without updating MicroShift using this update type.
The image you build can contain other updates as needed.

To begin a MicroShift update by embedding the new version in a RHEL for Edge image, use the procedures in the following documentation:

Applying updates on a RHEL for Edge system

To understand more about greenboot, see the following documentation:

About updates using RPMs

Updating {product-title} for non-image-based Red Hat Enterprise Linux (RHEL) systems requires updating the RPMs.

For patch releases, such as 4.22.1 to 4.22.2, simply update the RPMs.
For minor-version release updates, add the step of enabling the compatible update repository by using your subscription manager.

You can back up application data as needed and move the data copy to a secure location when using any update type.

Apply patch updates using RPMs

Updating MicroShift on non rpm-ostree systems such as Red Hat Enterprise Linux (RHEL) requires downloading then updating the RPMs. For example, use the following procedure to upgrade from 4.22.0 to 4.22.1.

Keeping component versions in a supported configuration of Red Hat Device Edge can require updating MicroShift and RHEL at the same time. Ensure that your version of RHEL is compatible with the version of MicroShift you are updating to, especially if you are updating MicroShift across two minor versions. Otherwise, you can create an unsupported configuration, break your node, or both. For more information, see the following link:

Red Hat Device Edge release compatibility matrix

You cannot downgrade MicroShift with this process. Downgrades are not supported.

Prerequisites

The system requirements for installing MicroShift have been met.
You have root user access to the host.
The version of MicroShift you have is compatible to upgrade to the version you are preparing to use.
You have verified that your host operating system is compatible with the version of MicroShift you are preparing to install.
You have completed a system backup.

Procedure

Update the MicroShift RPMs by running the following command:
```
$ sudo dnf update microshift
```
Restart MicroShift by running the following command:
```
$ sudo systemctl restart microshift
```
The greenboot system health check runs on this update type, but does not perform any actions. If the update fails, an error message appears with the instruction to check the logs.

About updates using RPMs

Updating {product-title} for non-image-based Red Hat Enterprise Linux (RHEL) systems requires updating the RPMs.

For patch releases, such as 4.22.1 to 4.22.2, simply update the RPMs.
For minor-version release updates, add the step of enabling the compatible update repository by using your subscription manager.

You can back up application data as needed and move the data copy to a secure location when using any update type.

Apply minor-version updates with RPMs

Updating a MicroShift minor version on non rpm-ostree systems such as Red Hat Enterprise Linux (RHEL) requires downloading then updating the RPMs. For example, use the following procedure to update from 4.18 to 4.20.

Red Hat Device Edge release compatibility matrix

You cannot downgrade MicroShift with this process. Downgrades are not supported.

Prerequisites

The system requirements for installing MicroShift have been met.
You have root user access to the host.
The version of MicroShift you have is compatible to upgrade to the version you are preparing to use.
You have verified that your host operating system is compatible with the version of MicroShift you are preparing to install.
You have completed a system backup.

Procedure

For all lifecycles, enable the repository for the release you want to update to by running the following command:

$ sudo subscription-manager repos \
    --enable rhocp-4.22-for-rhel-9-$(uname -m)-rpms \
    --enable fast-datapath-for-rhel-9-$(uname -m)-rpms

For extended support (EUS) releases, also enable the EUS repositories by running the following command:

$ sudo subscription-manager repos \
    --enable rhel-9-for-$(uname -m)-appstream-eus-rpms \
    --enable rhel-9-for-$(uname -m)-baseos-eus-rpms

Avoid unintended future updates into an unsupported configuration by locking your operating system version with the following command:
```
$ sudo subscription-manager release --set=9.8
```
Update the MicroShift RPMs by running the following command:
```
$ sudo dnf update microshift
```
Reboot the host to apply updates by running the following command:
```
$ sudo systemctl reboot
```
The system health check runs on this update type, but does not perform any actions. If the update fails, an error message appears with the instruction to check the logs.

Verification

Check if the health checks exited with a successful boot by running the following command:
```
$ sudo systemctl status greenboot-healthcheck
```
Check the health check logs by running the following command:
```
$ sudo journalctl -u greenboot-healthcheck
```

Migrate MicroShift to image mode for RHEL

Migrating MicroShift from a Red Hat Enterprise Linux for Edge (RHEL for Edge) system to a image mode for RHEL system requires building a new image mode for RHEL image containing the required version of MicroShift and any associated optional RPMs.

See the Red Hat Enterprise Linux (RHEL) documentation for general instructions on migrating RHEL for Edge systems to image mode for RHEL using the bootc switch command. Plan the upgrade process carefully. The following tips apply:

Follow the instructions in the RHEL documentation for converting rpm-ostree blueprint files to image mode container files.
You can use the rpm-ostree compose container-encapsulate image-compose command to create a base container image that can be used for bootc container builds. Then you can derive and familiarize yourself with an image mode for RHEL image that is based on existing ostree commits.
To fully adopt image mode for RHEL, define a container build pipeline.
Plan for UID and GID drift because RHEL for Edge and image mode for RHEL are not derived from the same parent image. See the RHEL documentation for more information.

Migrate MicroShift from RHEL for Edge to image mode for RHEL

Starting with MicroShift 4.19, you can migrate your MicroShift node from RHEL for Edge to image mode for RHEL if the final versions are a supported configuration of Red Hat Device Edge. Check compatibilities before beginning a migration. See the RHEL documentation for instructions to migrate your image-based RHEL system.

Work around UID and GID drift when migrating to image mode for RHEL

If you do not re-install operating systems that are running MicroShift, you must use a workaround for a possible UID and GID drift during the migration process. One way to solve this problem is to add systemd units that apply the necessary fixes before the affected system services are started.

Prerequisites

You have an existing RHEL for Edge deployment running MicroShift.
You have root access to the build host.
You have an image that you want to deploy.

Procedure

Solve the potential UID or GID drift for the Open vSwitch (OVS) systemd service, ovsdb-server.service, by adding the following command to the MicroShift image-build procedure:

# Install systemd configuration drop-ins to fix potential permission problems when upgrading from rpm-ostree commits to image mode container layers
RUN mkdir -p /usr/lib/systemd/system/ovsdb-server.service.d && \
    cat > /usr/lib/systemd/system/ovsdb-server.service.d/microshift-ovsdb-ownership.conf <<'EOF'
# The openvswitch database files must be owned by the appropriate user and its primary group. That the user and its group can be overwritten, recreate them.
[Service]
ExecStartPre=/bin/sh -c '/bin/getent passwd openvswitch >/dev/null || useradd -r openvswitch'
ExecStartPre=/bin/sh -c '/bin/getent group hugetlbfs >/dev/null || groupadd -r hugetlbfs'
ExecStartPre=/sbin/usermod -a -G hugetlbfs openvswitch
ExecStartPre=/bin/chown -Rhv openvswitch. /etc/openvswitch
EOF

After the MicroShift migration to image mode for RHEL is complete, this workaround is not needed and can be removed.

Verify system health

After you update MicroShift on Red Hat Enterprise Linux (RHEL) or image mode for RHEL, you can check the results of the update to verify the health of your system.

Check the status of greenboot health checks

You can check the status of greenboot health checks before making changes to the system or while troubleshooting. By using helpful commands to verify that greenboot scripts have finished running.

Procedure

Check the current greenboot health check status by running the following command:
```
$ systemctl show --property=SubState --value greenboot-healthcheck.service
```
where:

start

Greenboot checks are still running.

exited

Checks have passed and greenboot has exited. Greenboot runs the scripts in the green.d directory when the system is in a healthy state.

failed

Checks have not passed. Greenboot runs the scripts in the red.d directory when the system is in this state and restarts the system.
Check the numerical exit code of the greenboot health check service by running the following command:
```
$ systemctl show --property=ExecMainStatus --value greenboot-healthcheck.service
```
An exit code of 0 means the health check succeeded. A non-zero exit code means the health check failed.
To see a report showing a message about boot status, such as Boot Status is GREEN - Health Check SUCCESS, use the following command:
```
$ cat /run/motd.d/boot-status
```
Example output
```
Boot Status is GREEN - Health Check SUCCESS
```

Troubleshoot update problems

If an update process fails for MicroShift on RHEL for Edge or for a manual RPM update, you can check update paths, review journal and greenboot health check logs, and use other techniques to diagnose and resolve the problem.

Troubleshoot MicroShift updates

In some cases, MicroShift might fail to update. In these events, it is helpful to understand failure types and how to troubleshoot them.

Update path is blocked by MicroShift version sequence

Non-EUS versions of MicroShift require serial updates. For example, if you attempt to update from MicroShift 4.15.5 directly to 4.17.1, the update fails. You must first update 4.15.5 to 4.16.z, and then you can update from 4.16.z to 4.17.0.

Update path is blocked by version incompatibility

RPM dependency errors result if a MicroShift update is incompatible with the version of Red Hat Enterprise Linux for Edge (RHEL for Edge) or Red Hat Enterprise Linux (RHEL). For more information, see "Red Hat Device Edge release compatibility matrix".

RHEL for Edge update failed

If you updated on an rpm-ostree system, the greenboot health check automatically logs and acts on system health. A system rollback by greenboot can indicate an update failure. In cases where the update failed, but greenboot did not complete a system rollback, you can troubleshoot using the RHEL for Edge documentation linked in the "Additional resources" section.

Manually check the greenboot logs to verify system health by running the following command:

$ sudo systemctl restart --no-block greenboot-healthcheck && sudo journalctl -fu greenboot-healthcheck

Manual RPM update failed

If you updated by using RPMs on a non-OSTree system, greenboot can indicate an update failure, but the health checks are only informative. Checking the system logs is the next step in troubleshooting a manual RPM update failure. You can use greenboot and the sos report tool to check both the MicroShift update and the host system.

Check journal logs after updates

You can use journal logs to help diagnose MicroShift update failures. The default configuration of the systemd journal service stores data in a volatile directory, which does not persist across restarts. To retain logs across restarts, enable log persistence and set a maximum size limit for journal data.

Procedure

Get comprehensive MicroShift journal logs by running the following command:
```
$ sudo journalctl -u microshift
```
Check the greenboot journal logs by running the following command:
```
$ sudo journalctl -u greenboot-healthcheck
```
Examining the comprehensive logs of a specific boot uses three steps. First list the boots, then select the one you want from the list you obtained:
- List the boots present in the journal logs by running the following command:
  $ sudo journalctl --list-boots
  Example output
  IDX BOOT ID FIRST ENTRY LAST ENTRY 0 681ece6f5c3047e183e9d43268c5527f <Day> <Date> 12:27:58 UTC <Day> <Date>> 13:39:41 UTC #....
- Check the journal logs for the specific boot by running the following command:
  $ sudo journalctl --boot _<idx_or_boot_id>
  where:
  
  idx_or_boot_id
  
  Replace <idx_or_boot_id> with the IDX or the BOOT ID number assigned to the specific boot that you want to check.
- Check the journal logs for the boot of a specific service by running the following command:
  $ sudo journalctl --boot <idx_or_boot_id> -u <service_name>
  where:
  
  idx_or_boot_id
  
  Replace <idx_or_boot_id> with the IDX or the BOOT ID number assigned to the specific boot that you want to check.
  
  service_name
  
  Replace <service_name> with the name of the service that you want to check.

Identify potential update issues

Before you upgrade MicroShift, you can check which processes and configurations might be affected by the update so that you can plan remediations and minimize downtime.

Red Hat Device Edge release compatibility matrix

Red Hat Enterprise Linux (RHEL) and MicroShift work together as a single solution for device-edge computing. You can update each component separately, but the product versions must be compatible.

Supported configurations of Red Hat Device Edge use verified releases for each together as listed in the following table:

Be sure to check the support status of a release on the product lifecycle page.

RHEL Version(s)	MicroShift Version	Supported MicroShift Version → Version Updates
10.2	4.22	4.22.0 → 4.22.z (Technology Preview)
9.8	4.22	4.22.0 → 4.22.z, 4.22 on RHEL 9.8 → 4.22 on RHEL 10.2 (Technology Preview)
9.6	4.21	4.21.0 → 4.21.z, 4.21 → 4.22, 4.21 → 4.22 on RHEL 9.8, 4.21 → 4.22 on RHEL 10.2 (Technology Preview)
9.6	4.20	4.20.0 → 4.20.z, 4.20 → 4.21, 4.20 → 4.22 on RHEL 10.2 (Technology Preview)
9.6	4.19	4.19.0 → 4.19.z, 4.19 → 4.20
9.4	4.18	4.18.0 → 4.18.z, 4.18 → 4.20 on RHEL 9.6
9.4	4.17	4.17.1 → 4.17.z, 4.17 → 4.18
9.4	4.16	4.16.0 → 4.16.z, 4.16 → 4.17, 4.16 → 4.18

Migrate from RHEL 9 to RHEL 10

To migrate MicroShift from an existing Red Hat Enterprise Linux for Edge (RHEL for Edge) system running 9.8 to one running 10.2, you must embed MicroShift into a new operating system image. This process transitions your deployment from an rpm-ostree-based system to an image mode for RHEL (bootc) system, and requires planning for any UID or GID drift that may occur.

Migrate MicroShift to image mode for RHEL

Follow the instructions in the RHEL documentation for converting rpm-ostree blueprint files to image mode container files.
You can use the rpm-ostree compose container-encapsulate image-compose command to create a base container image that can be used for bootc container builds. Then you can derive and familiarize yourself with an image mode for RHEL image that is based on existing ostree commits.
To fully adopt image mode for RHEL, define a container build pipeline.
Plan for UID and GID drift because RHEL for Edge and image mode for RHEL are not derived from the same parent image. See the RHEL documentation for more information.

Work around UID and GID drift when migrating to image mode for RHEL

Prerequisites

You have an existing RHEL for Edge deployment running MicroShift.
You have root access to the build host.
You have an image that you want to deploy.

Procedure

Solve the potential UID or GID drift for the Open vSwitch (OVS) systemd service, ovsdb-server.service, by adding the following command to the MicroShift image-build procedure:

# Install systemd configuration drop-ins to fix potential permission problems when upgrading from rpm-ostree commits to image mode container layers
RUN mkdir -p /usr/lib/systemd/system/ovsdb-server.service.d && \
    cat > /usr/lib/systemd/system/ovsdb-server.service.d/microshift-ovsdb-ownership.conf <<'EOF'
# The openvswitch database files must be owned by the appropriate user and its primary group. That the user and its group can be overwritten, recreate them.
[Service]
ExecStartPre=/bin/sh -c '/bin/getent passwd openvswitch >/dev/null || useradd -r openvswitch'
ExecStartPre=/bin/sh -c '/bin/getent group hugetlbfs >/dev/null || groupadd -r hugetlbfs'
ExecStartPre=/sbin/usermod -a -G hugetlbfs openvswitch
ExecStartPre=/bin/chown -Rhv openvswitch. /etc/openvswitch
EOF

After the MicroShift migration to image mode for RHEL is complete, this workaround is not needed and can be removed.

Update existing Kubernetes storage objects

When you update MicroShift to a new version, existing stored Kubernetes objects may need to be migrated to their latest API storage version. The storage version migration process updates the stored representation of resources to match the API version expected by the new release, ensuring that your cluster remains fully operational after the update.

Updating stored data to the latest storage version

To update stored data to the latest Kubernetes storage version, perform a storage migration.

The procedure shows an example of converting existing objects on the v1beta1 version to the current version, such as v1beta2, to ensure compatibility with the cluster APIs.

Procedure

Either you or any controller that has support for the StorageVersionMigration API must trigger a migration request. Use the following example request for reference:
Example request
```
apiVersion: migration.k8s.io/v1alpha1
kind: StorageVersionMigration
metadata:
  name: v1beta1
spec:
  resource:
    group: example.storage.k8s.io
    resource: volumeclasses
    version: v1alpha1
# ...
```
where:

resource.resource

Specifies the plural name of the resource.

resource.version

Specifies the version to update to.

Verification

To monitor the progress of the update, review the status of the StorageVersionMigration custom resource (CR).

A migration fails when you misname a group or resource. Incompatible versions between the previous and latest versions can also cause a migration to fail.

Storage migration failure

Storage migration failures typically result from incompatible changes to custom resources (CRs) between MicroShift versions. If a storage migration fails, the CR versions are likely incompatible and require manual review.