Administer

About backing up and restoring MicroShift data

Backing up and restoring MicroShift data applies to the database only, and not to any application data. Before you can create a manual backup, greenboot health checks must finish running and you must stop the MicroShift service.

  • On rpm-ostree systems, MicroShift automatically creates a backup on every start. These automatic backups are deleted and replaced with the latest backup each time the system restarts.

  • Data is also automatically restored on an rpm-ostree system after a greenboot system rollback. This data restoration ensures that the database matches the software running on the host after the rollback is completed.

  • On other system types, you must back up and restore data manually.

Automated backups are in the /var/lib/microshift-backups directory by default. You can use this directory for manually backing up and restoring data by specifying it in each command. When you restore a backup, you must use the entire file path.

Note

The following procedures only backup and restore MicroShift data. Application data is not included.

Stop the MicroShift service

When you want to stop the MicroShift service, you must stop both the service and any deployed workloads.

Prerequisites

  • The MicroShift service is running.

Procedure

  1. Enter the following command to stop the MicroShift service:

    $ sudo systemctl stop microshift

  2. Workloads deployed on MicroShift might continue running even after the MicroShift service has been stopped. Enter the following command to display running workloads:

    $ sudo crictl ps -a

  3. Enter the following commands to stop the deployed workloads:

    $ sudo systemctl stop kubepods.slice

Back up MicroShift data manually

To back up {product-title} data manually, you can run microshift backup with a full path to the backup location. Stop the service first and use the entire path for the output file or directory.

You can back up MicroShift data manually at any time. Back up your data before system updates to preserve it for use if an update fails or for other system trouble. You can use the /var/lib/microshift-backups for manually backing up and restoring data by specifying it in each command.

Prerequisites

  • You have root access to the host.

  • MicroShift is stopped.

Procedure

  1. Manually create a backup by using the parent directory and specifying a name, such as /var/lib/microshift-backups/<manual_backup>, by running the following command:

    $ sudo microshift backup /var/lib/microshift-backups/<manual_backup>

    • For <manual_backup>, specify the backup name that you want to use.

      Example output
      ??? I1017 07:38:16.770506    5900 data_manager.go:92] "Copying data to backup directory" storage="/var/lib/microshift-backups" name="test" data="/var/lib/microshift"
??? I1017 07:38:16.770713    5900 data_manager.go:227] "Starting copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift /var/lib/microshift-backups/test"
??? I1017 07:38:16.776162    5900 data_manager.go:241] "Finished copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift /var/lib/microshift-backups/test"
??? I1017 07:38:16.776256    5900 data_manager.go:125] "Copied data to backup directory" backup="/var/lib/microshift-backups/test" data="/var/lib/microshift"

  2. Optional: Manually create a backup in a specific parent directory with a custom name by running the following command:

    $ sudo microshift backup /mnt/<other_backups_location>/<another_manual_backup>

    • For <other_backups_location>, specify the directory that you want to use.

    • For <another_manual_backup>, specify the backup name that you want to use.

Verification

  • You can verify that the backup exists by viewing the data in the directory you chose. For example, /var/lib/microshift-backups/<manual_backup>/ or /mnt/<other_backups_location>/<another_manual_backup>.

Restore MicroShift data backups manually

To restore {product-title} data after an update or data loss, you can run microshift restore with the full path to the backup. Backups can be restored after updates, or after other system events that remove or damage required data. When you restore a backup, you must use the entire file path.

Note

On an rpm-ostree system, MicroShift backs up and restores data automatically. Automated backups are in the /var/lib/microshift-backups directory by default.
Prerequisites

  • Root access to the host.

  • You have the full path of the data backup file.

  • The MicroShift service is stopped.

Procedure

  1. Manually restore MicroShift data by using the full file path of the backup you want to restore by running the following command:

    $ sudo microshift restore /var/lib/microshift-backups/<manual_backup>

    • For <manual_backup>, specify the backup name that you want to use. Optionally, you can also restore automatic ostree backups using the full file path.

      Example output
      ??? I1017 07:39:52.055165    6007 data_manager.go:131] "Copying backup to data directory" storage="/var/lib/microshift-backups" name="test" data="/var/lib/microshift"
??? I1017 07:39:52.055243    6007 data_manager.go:154] "Renaming existing data dir" data="/var/lib/microshift" renamedTo="/var/lib/microshift.saved"
??? I1017 07:39:52.055326    6007 data_manager.go:227] "Starting copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift-backups/test /var/lib/microshift"
??? I1017 07:39:52.061363    6007 data_manager.go:241] "Finished copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift-backups/test /var/lib/microshift"
??? I1017 07:39:52.061404    6007 data_manager.go:175] "Removing temporary data directory" path="/var/lib/microshift.saved"
??? I1017 07:39:52.063745    6007 data_manager.go:180] "Copied backup to data directory" name="test" data="/var/lib/microshift"

  2. Optional. Manually restore data from a customized directory by using the full file path of the backup. Run the following command:

    $ sudo microshift restore /mnt/<other_backups_location>/<another_manual_backup>

    • For <other_backups_location>, specify the directory that you used.

    • For <another_manual_backup>, specify the backup name that you used when creating the backup you are restoring.

  3. Restart the host. Restarting the host enables all workloads and pods to restart.

Verification

  • Use the oc get pods -A command to verify that the node is running, then check the restored data.

    $ oc get pods -A
    Example output
    NAMESPACE                   NAME                                                     READY   STATUS   RESTARTS  AGE
default                     i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running  0		    46m
kube-system                 csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running  0		    51m
openshift-dns               dns-default-45jl7                                        2/2     Running  0		    50m
openshift-dns               node-resolver-7wmzf                                      1/1     Running  0		    51m
openshift-ingress           router-default-78b86fbf9d-qvj9s                          1/1     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-master-5rfhh                                     4/4     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-node-gcnt6                                       1/1     Running  0		    51m
openshift-service-ca        service-ca-bf5b7c9f8-pn6rk                               1/1     Running  0		    51m
openshift-storage           topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running  0		    51m
openshift-storage           topolvm-node-rht2m                                       3/3     Running  0		    50m
    Note

    This example output shows a basic MicroShift installation. If you installed optional RPMs, the status of pods running those services is displayed in the output.

Modify backup and restore commands to automate data recovery

The --auto-recovery option stores {product-title} backups in one directory and selects the latest backup when you restore. You add the option to your backup and restore commands for automatic recovery.

The --auto-recovery option treats the PATH argument as a path to a directory that holds all the backups for automated recovery, and not just as a path to a particular backup file. You can use the --auto-recovery option with both backup and restore commands.

  • For example, if you use the automatic recovery option with restore, such as in microshift restore --auto-recovery PATH, running the modified command automatically selects and restores the most recent backup.

  • If you use the same option in the microshift backup command, such as in microshift backup --auto-recovery PATH, a new backup is created in the PATH.

  • By default, microshift restore --auto-recovery PATH creates a backup of the failed MicroShift data in PATH/failed. You can add the --dont-save-failed option to disable the creation of failed backup data.

Important

You can only use the --dont-save-failed option with the restore command.

Create backups using the auto-recovery feature

To create backups for {product-title} automatic recovery, you can run microshift backup --auto-recovery with a directory path. The command then stores each backup in that directory so that the latest is available when you restore.

Note

Creating backups requires stopping MicroShift. You must decide on the best time to stop MicroShift.
Prerequisites

  • You stopped MicroShift.

Procedure

  • Create and store backups in the directory you choose by running the following command:

    $ sudo microshift backup --auto-recovery <path_of_directory>

    • For <path_of_directory>, specify the path of the directory that stores backups. For example, /var/lib/microshift-auto-recovery.

      Note

      The --auto-recovery option modifies the interpretation of the PATH argument from the final backup path to a directory that holds all of the backups for automated recovery.
      Example output
      ??? I1104 09:18:52.100725    8906 system.go:58] "OSTree deployments" deployments=[{"id":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1","booted":true,"staged":false,"pinned":false},{"id":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","booted":false,"staged":false,"pinned":false}]
??? I1104 09:18:52.100895    8906 data_manager.go:83] "Copying data to backup directory" storage="/var/lib/microshift-auto-recovery" name="20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1" data="/var/lib/microshift"
??? I1104 09:18:52.102296    8906 disk_space.go:33] Calculated size of "/var/lib/microshift": 261M - increasing by 10% for safety: 287M
??? I1104 09:18:52.102321    8906 disk_space.go:44] Calculated available disk space for "/var/lib/microshift-auto-recovery": 1658M
??? I1104 09:18:52.105700    8906 atomic_dir_copy.go:66] "Made an intermediate copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift /var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1.tmp.99142"
??? I1104 09:18:52.105732    8906 atomic_dir_copy.go:115] "Renamed to final destination" src="/var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1.tmp.99142" dest="/var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"
??? I1104 09:18:52.105749    8906 data_manager.go:120] "Copied data to backup directory" backup="/var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1" data="/var/lib/microshift"
/var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1
Verification

  • Verify that the backup you created exists in your customized storage directory by running the following command:

    $ sudo ls -la <path_of_directory>

    • For <path_of_directory>, specify the path of the directory that stores backups. For example, /var/lib/microshift-auto-recovery.

Restore backups using the auto-recovery feature

To recover {product-title} data after loss or damage, you can run the microshift restore --auto-recovery command with your backups directory, which restores the latest backup. Previously restored backups that used automatic recovery are moved to your PATH/restored directory.

Prerequisites

  • You have stopped MicroShift.

Procedure

  1. Restore the latest backup from your backups directory by running the following command:

    $ sudo microshift restore --auto-recovery <path_of_directory>

    • For <path_of_directory>, specify the path of the directory that stores backups. For example, /var/lib/microshift-auto-recovery.

      Note

      • The --auto-recovery option copies the MicroShift data to /var/lib/microshift-auto-recovery/failed/ for later investigation, selects the most recent backup, and restores it.

      • The --dont-save-failed option disables the backing up of failed MicroShift data.
      Example output
      ??? I1104 09:19:28.617225    8950 state.go:80] "Read state from the disk" state={"LastBackup":"20241022101528_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"}
??? I1104 09:19:28.617323    8950 storage.go:78] "Auto-recovery backup storage read and parsed" dirs=["20241022101255_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","20241022101520_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","20241022101528_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1","restored"] backups=[{"CreationTime":"2024-10-22T10:12:55Z","Version":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"},{"CreationTime":"2024-10-22T10:15:20Z","Version":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"},{"CreationTime":"2024-10-22T10:15:28Z","Version":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"},{"CreationTime":"2024-11-04T09:18:52Z","Version":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"}]
??? I1104 09:19:28.617350    8950 storage.go:40] "Filtered list of backups - removed previously restored backup" removed="20241022101528_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0" newList=[{"CreationTime":"2024-10-22T10:12:55Z","Version":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"},{"CreationTime":"2024-10-22T10:15:20Z","Version":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"},{"CreationTime":"2024-11-04T09:18:52Z","Version":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"}]
??? I1104 09:19:28.633237    8950 system.go:58] "OSTree deployments" deployments=[{"id":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1","booted":true,"staged":false,"pinned":false},{"id":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","booted":false,"staged":false,"pinned":false}]
??? I1104 09:19:28.633258    8950 storage.go:49] "Filtered list of backups by version" version="default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1" newList=[{"CreationTime":"2024-11-04T09:18:52Z","Version":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"}]
??? I1104 09:19:28.633268    8950 restore.go:170] "Potential backups" bz=[{"CreationTime":"2024-11-04T09:18:52Z","Version":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"}]
??? I1104 09:19:28.633277    8950 restore.go:173] "Candidate backup for restore" b={"CreationTime":"2024-11-04T09:18:52Z","Version":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"}
??? I1104 09:19:28.634007    8950 disk_space.go:33] Calculated size of "/var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1": 261M - increasing by 10% for safety: 287M
??? I1104 09:19:28.634096    8950 disk_space.go:44] Calculated available disk space for "/var/lib": 1658M
??? I1104 09:19:28.634507    8950 disk_space.go:33] Calculated size of "/var/lib/microshift": 261M - increasing by 10% for safety: 287M
??? I1104 09:19:28.634522    8950 disk_space.go:44] Calculated available disk space for "/var/lib/microshift-auto-recovery": 1658M
??? I1104 09:19:28.649719    8950 system.go:58] "OSTree deployments" deployments=[{"id":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1","booted":true,"staged":false,"pinned":false},{"id":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","booted":false,"staged":false,"pinned":false}]
??? I1104 09:19:28.653880    8950 atomic_dir_copy.go:66] "Made an intermediate copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift /var/lib/microshift-auto-recovery/failed/20241104091928_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1.tmp.22742"
??? I1104 09:19:28.657362    8950 atomic_dir_copy.go:66] "Made an intermediate copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1 /var/lib/microshift.tmp.482"
??? I1104 09:19:28.657385    8950 state.go:40] "Saving intermediate state" state="{\"LastBackup\":\"20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1\"}" path="/var/lib/microshift-auto-recovery/state.json.tmp.41544"
??? I1104 09:19:28.662438    8950 atomic_dir_copy.go:115] "Renamed to final destination" src="/var/lib/microshift.tmp.482" dest="/var/lib/microshift"
??? I1104 09:19:28.662451    8950 state.go:46] "Moving state file to final path" intermediatePath="/var/lib/microshift-auto-recovery/state.json.tmp.41544" finalPath="/var/lib/microshift-auto-recovery/state.json"
??? I1104 09:19:28.662521    8950 atomic_dir_copy.go:115] "Renamed to final destination" src="/var/lib/microshift-auto-recovery/failed/20241104091928_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1.tmp.22742" dest="/var/lib/microshift-auto-recovery/failed/20241104091928_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"
??? I1104 09:19:28.662969    8950 atomic_dir_copy.go:115] "Renamed to final destination" src="/var/lib/microshift-auto-recovery/20241022101528_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0" dest="/var/lib/microshift-auto-recovery/restored/20241022101528_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"
??? I1104 09:19:28.662983    8950 restore.go:141] "Auto-recovery restore completed".
      Important

      • The restore command does not restart MicroShift after restoration. When you execute this command, MicroShift service has already failed or you stopped it.

      • MicroShift does not monitor the disk space of any filesystem. You must ensure that your automation handles old backup removal. For example, you can add this process to the auto-recovery service or add another service that runs periodically.

  2. Restart MicroShift by running the following command:

    $ sudo systemctl restart microshift
Verification

  • Verify that MicroShift has started successfully by running the following command:

    $ oc get pods -A
    Example output
    NAMESPACE                   NAME                                                     READY   STATUS   RESTARTS  AGE
default                     i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running  0		    46m
kube-system                 csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running  0		    51m
openshift-dns               dns-default-45jl7                                        2/2     Running  0		    50m
openshift-dns               node-resolver-7wmzf                                      1/1     Running  0		    51m
openshift-ingress           router-default-78b86fbf9d-qvj9s                          1/1     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-master-5rfhh                                     4/4     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-node-gcnt6                                       1/1     Running  0		    51m
openshift-service-ca        service-ca-bf5b7c9f8-pn6rk                               1/1     Running  0		    51m
openshift-storage           topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running  0		    51m
openshift-storage           topolvm-node-rht2m                                       3/3     Running  0		    50m
    Note

    This example output shows a basic MicroShift installation. If you installed optional RPMs, the status of pods running those services is displayed in the output.
Use automatic recovery in RPM systems

To use automatic recovery for {product-title} on RPM systems, you can create the 10-auto-recovery.conf file, the microshift-auto-recovery.service unit, and the microshift-auto-recovery script. Systemd runs the recovery service when the {product-title} service does not start, and the script restores the latest backup.

As a use case, consider the following example situation in which you want to automate the automatic recovery process for RPM systems that use the systemd service.

Procedure

  1. Create a directory for the microshift systemd service by running the following command:

    $ sudo mkdir -p /usr/lib/systemd/system/microshift.service.d

  2. To instruct systemd to run microshift-auto-recovery.service when the microshift.service fails, create the 10-auto-recovery.conf file by running the following command:

    $ sudo tee /usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf > /dev/null <<'EOF'
[Unit]
OnFailure=microshift-auto-recovery.service
StartLimitIntervalSec=25s

[Service]
RestartMode=direct
EOF

    • For StartLimitIntervalSec, specify a value greater than the default 10s for slower systems. A value that is too low can result in systemd never marking the microshift systemd service as failed, which means that the OnFailure= service does not get triggered.

    • RestartMode=direct prevents systemd from entering failed state on every restart attempt. This ensures OnFailure is triggered only after StartLimitBurst is exceeded, not on each failure. In systemd v254 (RHEL-10), OnFailure behavior changed to trigger on every failure instead of only when restart limits are reached. RestartMode=direct restores the v249 behavior. This setting is ignored on RHEL-9.6 (systemd v252) where it does not exist.

  3. Create the microshift-auto-recovery.service file by running the following command:

    $ sudo tee /usr/lib/systemd/system/microshift-auto-recovery.service > /dev/null <<'EOF'
[Unit]
Description=MicroShift auto-recovery

[Service]
Type=oneshot
ExecStart=/usr/bin/microshift-auto-recovery

[Install]
WantedBy=multi-user.target
EOF

  4. Create the microshift-auto-recovery script by running the following command:

    $ sudo tee /usr/bin/microshift-auto-recovery > /dev/null <<'EOF'
#!/usr/bin/env bash
set -xeuo pipefail

# If greenboot uses a non-default file for clearing boot_counter, use boot_success instead.
if grep -q  "/boot/grubenv" /usr/libexec/greenboot/greenboot-grub2-set-success; then
    if grub2-editenv - list | grep -q ^boot_success=0; then
        echo "Greenboot didn't decide the system is healthy after staging new deployment."
        echo "Quitting to not interfere with the process"
        exit 0
    fi
else
    if grub2-editenv - list | grep -q ^boot_counter=; then
        echo "Greenboot didn't decide the system is healthy after staging a new deployment."
        echo "Quitting to not interfere with the process"
        exit 0
    fi
fi

/usr/bin/microshift restore --auto-recovery /var/lib/microshift-auto-recovery
/usr/bin/systemctl reset-failed microshift
/usr/bin/systemctl start microshift

echo "DONE"
EOF

  5. Make the script executable by running the following command:

    $ sudo chmod +x /usr/bin/microshift-auto-recovery

  6. Reload the system configuration by running the following command:

    $ sudo systemctl daemon-reload
Use automatic recovery with RHEL for Edge

To use automatic recovery for {product-title} on RHEL for Edge systems, you can add the auto-recovery systemd service, 10-auto-recovery.conf, and the microshift-auto-recovery script to your blueprint. Use blueprint customizations so the image includes these files and recovery runs automatically.

Important

You must include the entire auto-recovery process for RHEL for Edge systems that use systemd in the blueprint file.
Prerequisites

  • You installed Podman.

  • You installed the command-line composer-cli tool.

Procedure

  1. Optional: Because the composer-cli can only create files in the /etc directory, package your files into an RPM that you include the blueprint.

  2. Use the following example to create your blueprint file:

    [[customizations.directories]]
path = "/etc/systemd/system/microshift.service.d"

[[customizations.directories]]
path = "/etc/bin"

[[customizations.files]]
path = "/etc/systemd/system/microshift.service.d/10-auto-recovery.conf"
data = """
[Unit]
OnFailure=microshift-auto-recovery.service
"""

[[customizations.files]]
path = "/etc/systemd/system/microshift-auto-recovery.service"
data = """
[Unit]
Description=MicroShift auto-recovery
[Service]
Type=oneshot
ExecStart=/etc/bin/microshift-auto-recovery
[Install]
WantedBy=multi-user.target
"""

[[customizations.files]]
path = "/etc/bin/microshift-auto-recovery"
mode = "0755"
data = """
#!/usr/bin/env bash
set -xeuo pipefail
# If greenboot uses a non-default file for clearing boot_counter, use boot_success instead.
if grep -q  "/boot/grubenv" /usr/libexec/greenboot/greenboot-grub2-set-success; then
    if grub2-editenv - list | grep -q ^boot_success=0; then
        echo "Greenboot didn't decide the system is healthy after staging a new deployment."
        echo "Quitting to not interfere with the process"
        exit 0
    fi
else
    if grub2-editenv - list | grep -q ^boot_counter=; then
        echo "Greenboot didn't decide the system is healthy after staging a new deployment."
        echo "Quitting to not interfere with the process"
        exit 0
    fi
fi
/usr/bin/microshift restore --auto-recovery /var/lib/microshift-auto-recovery
/usr/bin/systemctl reset-failed microshift
/usr/bin/systemctl start microshift
echo "DONE"
"""

  3. For the next steps, see Preparing for image building.

Use automatic recovery in image mode for RHEL systems

To use automatic recovery for {product-title} on RHEL image-based systems, you can embed the 10-auto-recovery.conf and microshift-auto-recovery.service files in your Containerfile and rebuild the bootc image.

Important

You must include the entire auto-recovery process for image mode for RHEL systems that use systemd in the container file.
Prerequisites

  • You created a Containerfile as instructed in Building the bootc image.

  • You created the 10-auto-recovery.conf and microshift-auto-recovery.service files as explained in the "Using auto-recovery in RPM systems" section.

    Important

    The location of the 10-auto-recovery.conf and microshift-auto-recovery.service files must be relative to the Containerfile.

    For example, if the path to the Containerfile is /home/microshift/my-build/Containerfile, the systemd files need to be adjacent for proper embedding. The following paths are correct for this example:

    • /home/microshift/my-build/auto-rec/10-auto-recovery.conf

    • /home/microshift/my-build/auto-rec/microshift-auto-recovery.service

    • /home/microshift/my-build/auto-rec/microshift-auto-recovery

  • You created the microshift-auto-recovery script as explained in the "Using auto-recovery in RPM systems" section.

Procedure

  1. Use the following example snippet to update the container file that you use to prepare the image mode for RHEL image.

    RUN mkdir -p /usr/lib/systemd/system/microshift.service.d
COPY ./auto-rec/10-auto-recovery.conf /usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf
COPY ./auto-rec/microshift-auto-recovery.service /usr/lib/systemd/system/
COPY ./auto-rec/microshift-auto-recovery /usr/bin/
RUN chmod +x /usr/bin/microshift-auto-recovery
    Important

    Podman uses the host subscription information and repositories inside the container when building the container image. If the rhocp and fast-datapath repositories are not available on the host, the build fails.

  2. Rebuild your local bootc image by running the following image build command:

    PULL_SECRET=~/.pull-secret.json
USER_PASSWD=<your_redhat_user_password>
IMAGE_NAME=microshift-4.18-bootc

sudo podman build --authfile "${PULL_SECRET}" -t "${IMAGE_NAME}" \
    --build-arg USER_PASSWD="${USER_PASSWD}" \
    -f Containerfile
    Note

    Secrets are used during the image build in the following ways:

    • The podman --authfile argument is required to pull the base rhel-bootc:9.4 image from the registry.redhat.io registry.

    • The build USER_PASSWD argument is used to set a password for the redhat user.
Verification

  • Verify that the local bootc image was created by running the following command:

    $ sudo podman images "${IMAGE_NAME}"
    Example output
    REPOSITORY                       TAG         IMAGE ID      CREATED        SIZE
localhost/microshift-4.18-bootc  latest      193425283c00  2 minutes ago  2.31 GB

Troubleshoot data backup and restore

If you encounter issues when backing up or restoring MicroShift data, you can use the following information to diagnose and resolve common backup failures, restore failures, and storage migration problems.

Data backup failure

Data backups are automatic on rpm-ostree systems. If you are not using an rpm-ostree system and attempted to create a manual backup, certain conditions can cause the backup to fail.

MicroShift was stopped too soon after the system started

Wait until the system completes health checks and background processes before stopping MicroShift.

MicroShift stopped because of an error

Verify that MicroShift is healthy and in a running state before you create a backup.

Insufficient storage space

Verify that sufficient storage is available for MicroShift data before you create a backup.

Insufficient user permissions

Verify that you have the correct user permissions and configurations required to create a backup.

Check backup logs

Backup logs can help you identify the location and status of manual and automatic backups, and the processes that occurred during each backup.

  • Manual backup logs are displayed in the terminal output.

  • Automatic backup logs for rpm-ostree systems are available in the MicroShift journal logs.

Procedure

  • Check the journal logs:

    $ sudo journalctl -u microshift

Data restoration failure

The restoration of data can fail for many reasons, including storage and permission issues. Mismatched data versions can cause failures when MicroShift restarts.

Image-based systems data restore failed

Data restorations are automatic on rpm-ostree systems, but can fail, for example:

  • The only backups that are restored on rpm-ostree systems are backups from the current deployment or a rollback deployment. Backups are not taken on an unhealthy system.

    • Only the latest backups that have corresponding deployments are retained. Outdated backups that do not have a matching deployment are automatically removed.

    • Data is usually not restored from a newer version of MicroShift.

    • Ensure that the data you are restoring follows same versioning pattern as the update path. For example, if the destination version of MicroShift is an older version than the version of the MicroShift data you are currently using, the restoration can fail.

RPM-based manual data restore failed

If you are using an RPM system that is not rpm-ostree and tried to restore a manual backup, the following reasons can cause the restoration to fail:

  • If MicroShift stopped running because of an error, you cannot restore data.

    • Make sure the system is healthy.

    • Start it in a healthy state before attempting to restore data.

  • If you do not have enough storage space allocated for the incoming data, the restoration fails.

    • Make sure that your current system storage is configured to accept the restored data.

  • You are attempting to restore data from a newer version of MicroShift.

    • Ensure that the data you are restoring follows same versioning pattern as the update path. For example, if the destination version of MicroShift is an older version than the version of the MicroShift data you are attempting to use, the restoration can fail.

Storage migration failure

Storage migration failures typically result from incompatible changes to custom resources (CRs) between MicroShift versions. If a storage migration fails, the CR versions are likely incompatible and require manual review.

Check the installed version

You can check which version of MicroShift is running on your system by using the command-line interface, the API, or by checking the etcd version.

Check the version using the command-line interface

To begin troubleshooting, you must know your MicroShift version. One way to get this information is by using the command-line interface (CLI).

Procedure

  • Check the version information by running the following command:

    $ microshift version
    Example output
    MicroShift Version: 4.22-0.microshift-e6980e25
Base OCP Version: 4.22

Check the MicroShift version using the API

To begin troubleshooting, you must know your MicroShift version. One way to get this information is by using the API.

Procedure

  • To get the version number using the OpenShift CLI (oc), view the kube-public/microshift-version config map by running the following command:

    $ oc get configmap -n kube-public microshift-version -o yaml
    Example output
    apiVersion: v1
data:
  major: "4"
  minor: "20"
  version: 4.20.0-0.microshift-fa441af87431
kind: ConfigMap
metadata:
  creationTimestamp: "2025-11-03T21:06:11Z"
  name: microshift-version
  namespace: kube-public

Check the etcd version

You can get the version information for the etcd database included with your MicroShift by using one or both of the following methods, depending on the level of information that you need.

Procedure

  • To display the base database version information, run the following command:

    $ microshift-etcd version
    Example output
    microshift-etcd Version: 4.20.0
Base etcd Version: 3.5.13

  • To display the full database version information, run the following command:

    $ microshift-etcd version -o json
    Example output
    {
  "major": "4",
  "minor": "20",
  "gitVersion": "4.20.0",
  "gitCommit": "140777711962eb4e0b765c39dfd325fb0abb3622",
  "gitTreeState": "clean",
  "buildDate": "2025-11-03T16:37:53Z",
  "goVersion": "go1.21.9"
  "compiler": "gc",
  "platform": "linux/amd64",
  "patch": "",
  "etcdVersion": "3.5.13"
}

Troubleshoot a node

You can troubleshoot a MicroShift node by checking the status of the node and its active pods to identify issues.

Check the status of a node

You can check the status of a MicroShift node or see active pods. You can choose to run any or all of the following commands to help you get the information you need to troubleshoot the node.

Procedure

  • Check the system status, which returns the node status, by running the following command:

    $ sudo systemctl status microshift

    If MicroShift fails to start, this command returns the logs from the earlier run.

    Example healthy output
    ● microshift.service - MicroShift
     Loaded: loaded (/usr/lib/systemd/system/microshift.service; enabled; preset: disabled)
     Active: active (running) since <day> <date> 12:39:06 UTC; 47min ago
   Main PID: 20926 (microshift)
      Tasks: 14 (limit: 48063)
     Memory: 542.9M
        CPU: 2min 41.185s
     CGroup: /system.slice/microshift.service
             └─20926 microshift run

<Month-Day> 13:23:06 i-06166fbb376f14a8b.<hostname> microshift[20926]: kube-apiserver I0528 13:23:06.876001   20926 controll>
<Month-Day> 13:23:06 i-06166fbb376f14a8b.<hostname> microshift[20926]: kube-apiserver I0528 13:23:06.876574   20926 controll>
# ...

  • Optional: Get comprehensive logs by running the following command:

    $ sudo journalctl -u microshift
    Note

    The default configuration of the systemd journal service stores data in a volatile directory, which does not persist across restarts. To retain logs across system restarts, enable log persistence and set a maximum size limit for journal data.

  • If MicroShift is running, check the status of active pods by entering the following command:

    $ oc get pods -A
    Example output
    NAMESPACE                   NAME                                                     READY   STATUS   RESTARTS  AGE
default                     i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running  0		    46m
kube-system                 csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running  0		    51m
openshift-dns               dns-default-45jl7                                        2/2     Running  0		    50m
openshift-dns               node-resolver-7wmzf                                      1/1     Running  0		    51m
openshift-ingress           router-default-78b86fbf9d-qvj9s                          1/1     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-master-5rfhh                                     4/4     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-node-gcnt6                                       1/1     Running  0		    51m
openshift-service-ca        service-ca-bf5b7c9f8-pn6rk                               1/1     Running  0		    51m
openshift-storage           topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running  0		    51m
openshift-storage           topolvm-node-rht2m                                       3/3     Running  0		    50m
    Note

    This example output shows a basic MicroShift installation. If you installed optional RPMs, the status of pods running those services is displayed in the output.

Troubleshoot installation issues

If you encounter issues during MicroShift installation, you can gather diagnostic data by generating an sos report to share with Red Hat Support.

Gather data from an sos report

You can create an sosreport archive about a failing {op-system-full} host that you can share with Red Hat support for troubleshooting.

Prerequisites

  • You must have the sos package installed.

  • You have root access to the host.

Procedure

  1. Log in to the failing host as a root user.

  2. Perform the debug report creation procedure by running the following command:

    $ microshift-sos-report
    Example output
    sosreport (version 4.5.1)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.o0sznf_8 and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

        Distribution Website : https://www.redhat.com/
        Commercial Support   : https://www.access.redhat.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.


 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/2   microshift      [Running: microshift]
  Starting 2/2   microshift_ovn  [Running: microshift microshift_ovn]
  Finishing plugins              [Running: microshift]

  Finished running plugins

Found 1 total reports to obfuscate, processing up to 4 concurrently

sosreport-microshift-rhel9-2023-03-31-axjbyxw :    Beginning obfuscation...
sosreport-microshift-rhel9-2023-03-31-axjbyxw :    Obfuscation completed

Successfully obfuscated 1 report(s)

Creating compressed archive...

A mapping of obfuscated elements is available at
	/var/tmp/sosreport-microshift-rhel9-2023-03-31-axjbyxw-private_map

Your sosreport has been generated and saved in:
	/var/tmp/sosreport-microshift-rhel9-2023-03-31-axjbyxw-obfuscated.tar.xz

 Size	444.14KiB
 Owner	root
 sha256	922e5ff2db25014585b7c6c749d2c44c8492756d619df5e9838ce863f83d4269

Please send this file to your support representative.

Troubleshoot updates

If MicroShift fails to update, you can diagnose the issue by checking update failure types, reviewing journal logs, and verifying greenboot health check status.

Troubleshoot MicroShift updates

In some cases, MicroShift might fail to update. In these events, it is helpful to understand failure types and how to troubleshoot them.

Update path is blocked by MicroShift version sequence

Non-EUS versions of MicroShift require serial updates. For example, if you attempt to update from MicroShift 4.15.5 directly to 4.17.1, the update fails. You must first update 4.15.5 to 4.16.z, and then you can update from 4.16.z to 4.17.0.

Update path is blocked by version incompatibility

RPM dependency errors result if a MicroShift update is incompatible with the version of Red Hat Enterprise Linux for Edge (RHEL for Edge) or Red Hat Enterprise Linux (RHEL). For more information, see "Red Hat Device Edge release compatibility matrix".

RHEL for Edge update failed

If you updated on an rpm-ostree system, the greenboot health check automatically logs and acts on system health. A system rollback by greenboot can indicate an update failure. In cases where the update failed, but greenboot did not complete a system rollback, you can troubleshoot using the RHEL for Edge documentation linked in the "Additional resources" section.

  • Manually check the greenboot logs to verify system health by running the following command:

    $ sudo systemctl restart --no-block greenboot-healthcheck && sudo journalctl -fu greenboot-healthcheck
Manual RPM update failed

If you updated by using RPMs on a non-OSTree system, greenboot can indicate an update failure, but the health checks are only informative. Checking the system logs is the next step in troubleshooting a manual RPM update failure. You can use greenboot and the sos report tool to check both the MicroShift update and the host system.

Check journal logs after updates

You can use journal logs to help diagnose MicroShift update failures. The default configuration of the systemd journal service stores data in a volatile directory, which does not persist across restarts. To retain logs across restarts, enable log persistence and set a maximum size limit for journal data.

Procedure

  • Get comprehensive MicroShift journal logs by running the following command:

    $ sudo journalctl -u microshift

  • Check the greenboot journal logs by running the following command:

    $ sudo journalctl -u greenboot-healthcheck

  • Examining the comprehensive logs of a specific boot uses three steps. First list the boots, then select the one you want from the list you obtained:

    • List the boots present in the journal logs by running the following command:

      $ sudo journalctl --list-boots
      Example output
      IDX  BOOT ID                          	FIRST ENTRY                 LAST ENTRY
 0   681ece6f5c3047e183e9d43268c5527f 	<Day> <Date> 12:27:58 UTC 	<Day> <Date>> 13:39:41 UTC
#....

    • Check the journal logs for the specific boot by running the following command:

      $ sudo journalctl --boot _<idx_or_boot_id>

      where:

      idx_or_boot_id

      Replace <idx_or_boot_id> with the IDX or the BOOT ID number assigned to the specific boot that you want to check.

    • Check the journal logs for the boot of a specific service by running the following command:

      $ sudo journalctl --boot <idx_or_boot_id> -u <service_name>

      where:

      idx_or_boot_id

      Replace <idx_or_boot_id> with the IDX or the BOOT ID number assigned to the specific boot that you want to check.

      service_name

      Replace <service_name> with the name of the service that you want to check.

Check the status of greenboot health checks

You can check the status of greenboot health checks before making changes to the system or while troubleshooting. By using helpful commands to verify that greenboot scripts have finished running.

Procedure

  • Check the current greenboot health check status by running the following command:

    $ systemctl show --property=SubState --value greenboot-healthcheck.service

    where:

    start

    Greenboot checks are still running.

    exited

    Checks have passed and greenboot has exited. Greenboot runs the scripts in the green.d directory when the system is in a healthy state.

    failed

    Checks have not passed. Greenboot runs the scripts in the red.d directory when the system is in this state and restarts the system.

  • Check the numerical exit code of the greenboot health check service by running the following command:

    $ systemctl show --property=ExecMainStatus --value greenboot-healthcheck.service

    An exit code of 0 means the health check succeeded. A non-zero exit code means the health check failed.

  • To see a report showing a message about boot status, such as Boot Status is GREEN - Health Check SUCCESS, use the following command:

    $ cat /run/motd.d/boot-status
    Example output
    Boot Status is GREEN - Health Check SUCCESS

Check audit logs

You can identify pod security admission violations in a workload by reviewing the MicroShift server audit logs.

Identify pod security violations through audit logs

You can identify pod security admission violations in a workload by viewing the server audit logs. To do this, you must access and parse audit logs to find these violations.

Prerequisites

  • You have installed the jq utility.

  • You have root access to the node.

Procedure

  1. Retrieve the node name by running the following command:

    $ NODE_NAME=$(oc get node -ojsonpath='{.items[0].metadata.name}')

  2. View the available audit logs by running the following command:

    $ oc adm node-logs ${NODE_NAME} --path=kube-apiserver/
    Example output
    rhel-94.lab.local audit-2024-10-18T18-25-41.663.log
rhel-94.lab.local audit-2024-10-19T11-21-29.225.log
rhel-94.lab.local audit-2024-10-20T04-16-09.622.log
rhel-94.lab.local audit-2024-10-20T21-11-41.163.log
rhel-94.lab.local audit-2024-10-21T14-06-10.402.log
rhel-94.lab.local audit-2024-10-22T06-35-10.392.log
rhel-94.lab.local audit-2024-10-22T23-26-27.667.log
rhel-94.lab.local audit-2024-10-23T16-52-15.456.log
rhel-94.lab.local audit-2024-10-24T07-31-55.238.log

  3. Parse the audit logs to find pod security violations by running the following command:

    $ oc adm node-logs ${NODE_NAME} --path=kube-apiserver/audit.log \
  | jq -r 'select((.annotations["pod-security.kubernetes.io/audit-violations"] != null) and (.objectRef.resource=="pods")) | .objectRef.namespace + " " + .objectRef.name + " " + .objectRef.resource' \
  | sort | uniq -c

Troubleshoot etcd

MicroShift runs etcd as a managed, separate process to store system state. To ensure optimal performance and resolve issues, as an administrator, you can observe system activity and enforce memory usage limits by using the MicroShift configuration file.

Observe and debug the MicroShift etcd server

Monitoring the etcd server is critical for maintaining system stability and diagnosing errors. You can gather journalctl logs to observe and debug the etcd server logs.

Prerequisites

  • The MicroShift service is running.

Procedure

  • To get the logs for etcd, run the following command:

    $ sudo journalctl -u microshift-etcd.scope
    Note

    MicroShift logs can be accessed separately from etcd logs using the journalctl -u microshift command.

Configure the memoryLimitMB value to set parameters for the etcd server

By default, etcd uses as much memory as necessary to handle the system load. On memory-constrained systems, limiting the amount of memory etcd uses might be necessary. Configure the memoryLimitMB parameter to restrict the memory consumption of the etcd server.

Procedure

  • Edit the /etc/microshift/config.yaml configuration file to set the memoryLimitMB value.

    etcd:
  memoryLimitMB: 128
    Note

    The minimum required value for memoryLimitMB on MicroShift is 128 MB. Values close to the minimum value are more likely to impact etcd performance. Lower limits increase the time etcd takes to respond to queries. If the limit is too low or etcd usage is high, queries might time out.
Verification

  1. Restart MicroShift to apply the changes by running the following command:

    $ sudo systemctl restart microshift

  2. Verify that the new memoryLimitMB value is in use by running the following command:

    $ systemctl show --property=MemoryHigh microshift-etcd.scope

Manage responsive restarts and security certificates

MicroShift depends on device IP addresses and system-wide clock settings to remain consistent during runtime. When these settings change, or when security certificates approach expiration, MicroShift handles responsive restarts and certificate rotations to maintain secure operation.

IP address changes or clock adjustments

MicroShift depends on device IP addresses and system-wide clock settings to remain consistent during its runtime. However, these settings might occasionally change on edge devices.

For example, DHCP or Network Time Protocol (NTP) updates can change times. When these changes occur, some MicroShift components might stop functioning properly. To mitigate this situation, MicroShift monitors the IP address and system time and restarts if either setting changes.

The threshold for a clock-driven restart is a time change of greater than 10 seconds in either direction. Small drifts during regular NTP service adjustments do not trigger a restart.

Security certificate lifetime

MicroShift certificates are digital certificates that secure communication with communication protocols such as HTTPS. They fall into two basic categories:

Short-lived certificates

Valid for one year. Most server or leaf certificates are short-lived.

Long-lived certificates

Valid for 10 years. For example, the client certificate for system:admin user authentication, or the kube-apiserver external serving certificate signer.

MicroShift restarts automatically depending on certificate age.

Certificate rotation

Certificates that are expired or close to their expiration dates must be rotated to ensure continued MicroShift operation. Certificate rotation can occur automatically.

When MicroShift restarts for any reason, certificates that are close to expiring are rotated. A certificate that expires soon, or has already expired, can also cause an automatic MicroShift restart to perform a rotation.

Important

If the rotated certificate is a MicroShift certificate authority (CA), all signed certificates are also rotated. If you created custom CAs, you must rotate them manually.
Short-term-certificate rotation

Short-term certificates that are expired or close to their expiration dates must be rotated to ensure continued MicroShift operation.

The following situations describe MicroShift actions during short-term-certificate lifetime:

No rotation

When a short-term certificate is up to 5 months old, no rotation occurs.

Rotation at restart

When a short-term certificate is 5 to 8 months old, it is rotated when MicroShift starts or restarts.

Automatic restart for rotation

When a short-term certificate is more than 8 months old, MicroShift can automatically restart to rotate and apply a new certificate.

Long-term-certificate rotation

Long-term certificates that are expired or close to their expiration dates must be rotated to ensure continued MicroShift operation.

The following situations describe MicroShift actions during long-term certificate lifetime:

No rotation

When a long-term certificate is up to 8.5 years old, no rotation occurs.

Rotation at restart

When a long-term certificate is 8.5 to 9 years old, it is rotated when MicroShift starts or restarts.

Automatic restart for rotation

When a long-term certificate is more than 9 years old, MicroShift might automatically restart so that it can rotate and apply a new certificate.

Clean up data with support

When working with Red Hat Support to resolve system conflicts, you can use the microshift-cleanup-data script to safely remove MicroShift data, configuration, network data, and custom certificates.

Data cleanup script overview

You can see the usage and list available options of the microshift-cleanup-data script by running the script without arguments. Running the script without arguments does not delete any data or stop the MicroShift service.

Procedure

  1. See the usage and list the available options of the microshift-cleanup-data script by entering the following command:

    Warning

    Some script operations are destructive and can cause data loss. Review the specific procedure for each argument for detailed warnings.

    		 
    $ microshift-cleanup-data
    Example output
    Stop all MicroShift services, also cleaning their data

Usage: microshift-cleanup-data <--all [--keep-images] | --ovn | --cert>
   --all         Clean all MicroShift and OVN data
   --keep-images Keep container images when cleaning all data
   --ovn         Clean OVN data only
   --cert        Clean certificates only

Clean all data and configuration

You can clean up all the MicroShift data and configuration by running the microshift-cleanup-data script.

When you run the script with the --all argument, you perform the following clean up actions:

  • Stop and disable all MicroShift services

  • Delete all MicroShift pods

  • Delete all container image storage

  • Reset network configuration

  • Delete the /var/lib/microshift data directory

  • Delete OVN-K networking configuration

Prerequisites

  • You are logged into MicroShift.

  • You have filed a support case.

Procedure

  1. Clean up all the MicroShift data and configuration by running the microshift-cleanup-data script with the --all argument, by entering the following command:

    Warning

    This option deletes all MicroShift data and user workloads. Use with caution.

    		 
    $ sudo microshift-cleanup-data --all
    Tip

    The script prompts you to confirm the operation. Enter 1 or Yes to continue. Any other entry cancels the cleanup.
    Example output when you continue the cleanup
    DATA LOSS WARNING: Do you wish to stop and clean ALL MicroShift data AND cri-o container workloads?
1) Yes
2) No
#? 1
Stopping MicroShift services
Disabling MicroShift services
Removing MicroShift pods
Removing crio image storage
Deleting the br-int interface
Killing conmon, pause and OVN processes
Removing MicroShift configuration
Removing OVN configuration
MicroShift service was stopped
MicroShift service was disabled
Cleanup succeeded
    Example output when you cancel the cleanup
    DATA LOSS WARNING: Do you wish to stop and clean ALL MicroShift data AND cri-o container workloads?
1) Yes
2) No
#? no
Aborting cleanup
    Important

    The microshift-cleanup-data script stops and disables the MicroShift service.

  2. Restart the MicroShift service by running the following command:

    $ sudo systemctl enable --now microshift

Clean all data and keep the container images

You can retain the MicroShift container images while cleaning all data by running the microshift-cleanup-data script with the --all and --keep-images arguments.

Keeping the container images helps speed up MicroShift restart after data clean up because the necessary container images are already present locally when you start the service.

When you run the script with the --all and --keep-images arguments, you perform the following clean up actions:

  • Stop and disable all MicroShift services

  • Delete all MicroShift pods

  • Reset network configuration

  • Delete the /var/lib/microshift data directory

  • Delete OVN-K networking configuration

Warning

This option deletes all MicroShift data and user workloads. Use with caution.
Prerequisites

  • You are logged into MicroShift.

  • You have filed a support case.

Procedure

  1. Clean up all data and user workloads when retaining the MicroShift container images by running the following command:

    $ sudo microshift-cleanup-data --all --keep-images
    Example output
    DATA LOSS WARNING: Do you wish to stop and clean ALL MicroShift data AND cri-o container workloads?
1) Yes
2) No
#? Yes
Stopping MicroShift services
Disabling MicroShift services
Removing MicroShift pods
Deleting the br-int interface
Killing conmon, pause and OVN processes
Removing MicroShift configuration
Removing OVN configuration
MicroShift service was stopped
MicroShift service was disabled
Cleanup succeeded

  2. Verify that the container images are still present by running the following command:

    $ sudo crictl images | awk '{print $1}'
    Example output
    IMAGE
quay.io/openshift-release-dev/ocp-v4.0-art-dev
quay.io/openshift-release-dev/ocp-v4.0-art-dev
quay.io/openshift-release-dev/ocp-v4.0-art-dev
quay.io/openshift-release-dev/ocp-v4.0-art-dev
quay.io/openshift-release-dev/ocp-v4.0-art-dev
quay.io/openshift-release-dev/ocp-v4.0-art-dev
quay.io/openshift-release-dev/ocp-v4.0-art-dev
quay.io/openshift-release-dev/ocp-v4.0-art-dev
quay.io/openshift-release-dev/ocp-v4.0-art-dev
quay.io/openshift-release-dev/ocp-v4.0-art-dev
registry.redhat.io/lvms4/topolvm-rhel9
registry.redhat.io/openshift4/ose-csi-external-provisioner
registry.redhat.io/openshift4/ose-csi-external-resizer
registry.redhat.io/openshift4/ose-csi-livenessprobe
registry.redhat.io/openshift4/ose-csi-node-driver-registrar
registry.redhat.io/ubi9
    Important

    The microshift-cleanup-data script stops and disables the MicroShift service.

  3. Restart the MicroShift service by running the following command:

    $ sudo systemctl enable --now microshift

Clean the OVN-Kubernetes data

Reset OVN-Kubernetes (OVN-K) network configurations by running the microshift-cleanup-data script.

When you run the script with the --ovn argument, you perform the following clean up actions:

  • Stop all MicroShift services

  • Delete all MicroShift pods

  • Delete the OVN-K networking configuration

Prerequisites

  • You are logged into MicroShift.

  • You have filed a support case.

Procedure

  1. Clean up the OVN-K data by running the microshift-cleanup-data script with the --ovn argument, by entering the following command:

    $ sudo microshift-cleanup-data --ovn
    Example output
    Stopping MicroShift services
Removing MicroShift pods
Killing conmon, pause and OVN processes
Removing OVN configuration
MicroShift service was stopped
Cleanup succeeded
    Important

    The microshift-cleanup-data script stops the MicroShift service.

  2. Restart the MicroShift service by running the following command:

    $ sudo systemctl start microshift

Clean custom certificates data

To recreate MicroShift custom certificates upon service restart, reset them by using the microshift-cleanup-data script.

When you run the script with the --cert argument, you perform the following clean up actions:

  • Stop all MicroShift services

  • Delete all MicroShift pods

  • Delete all MicroShift certificates

Prerequisites

  • You are logged into MicroShift.

  • You have filed a support case.

Procedure

  1. Clean up the MicroShift certificates by running the microshift-cleanup-data script with the --cert argument, by entering the following command:

    $ sudo microshift-cleanup-data --cert
    Example output
    Stopping MicroShift services
Removing MicroShift pods
Removing MicroShift certificates
MicroShift service was stopped
Cleanup succeeded
    Important

    Running the script stops the MicroShift service.

  2. Restart the MicroShift service by running the following command:

    $ sudo systemctl start microshift

Get support

If you encounter issues with MicroShift that you cannot resolve independently, you can access support resources including the Red Hat Knowledgebase, submit a support case, and gather diagnostic data to share with Red Hat Support.

Get support

Red Hat offers several support channels to help you troubleshoot issues and get the most from {product-title}.

From the Red Hat Customer Portal, you can:

  • Search or browse through the Red Hat Knowledgebase of articles and solutions about Red Hat products.

  • Submit a support case to Red Hat Support.

  • Access other product documentation.

To identify issues with your cluster, you can use Red Hat Lightspeed in OpenShift Cluster Manager. Red Hat Lightspeed provides details about issues and, if available, information about how to solve a problem.

To suggest improvements or report errors, give specific details such as the section name and {product-title} version.

To report an error or to improve our documentation, you can submit a Jira issue by using your Red Hat Jira account.

About the Red Hat Knowledgebase

The Red Hat Knowledgebase helps you get the most from Red Hat products and technologies.

It includes articles, product documentation, and videos that outline best practices for installing, configuring, and using Red Hat products. You can also search for solutions to known issues. Each solution has a root cause description and steps to fix the problem.

Additional resources

Search the Red Hat Knowledgebase

Search the Red Hat Knowledgebase to find solutions to known issues and resolve problems quickly without opening a support case.

Prerequisites

  • You have a Red Hat Customer Portal account.

Procedure

  1. Log in to the Red Hat Customer Portal.

  2. Click Search.

  3. In the search field, input keywords and strings relating to the problem, including:

    • {product-title} components (such as etcd)

    • Related procedure (such as installation)

    • Warnings, error messages, and other outputs related to explicit failures

  4. Click the Enter key.

  5. Optional: Select the {product-title} product filter.

  6. Optional: Select the Documentation content type filter.

Submit a support case

If you encounter issues with MicroShift that cannot be resolved through the standard troubleshooting, you can submit a support case to Red Hat. Providing detailed descriptions and diagnostic data helps Red Hat Support to analyze the problem and help you with a resolution.

Prerequisites

  • The MicroShift service is running.

  • You have installed the OpenShift CLI (oc).

  • You have a Red Hat Customer Portal account.

  • You have a Red Hat Standard or Premium subscription.

Procedure

  1. Log in to the Customer Support page of the Red Hat Customer Portal.

  2. Click Get support.

  3. On the Cases tab of the Customer Support page:

    1. Optional: Change the pre-filled account and owner details if needed.

    2. Select the appropriate category for your issue, such as Bug or Defect, and click Continue.

  4. Enter the following information:

    1. In the Summary field, enter a concise but descriptive problem summary and further details about the symptoms being experienced, as well as your expectations.

    2. Select Red Hat Device Edge from the Product drop-down menu.

    3. Select 4 from the Version drop-down.

  5. Review the list of suggested Red Hat Knowledgebase solutions for a potential match against the problem that is being reported. If the suggested articles do not address the issue, click Continue.

  6. Review the updated list of suggested Red Hat Knowledgebase solutions for a potential match against the problem that is being reported. The list is refined as you provide more information during the case creation process. If the suggested articles do not address the issue, click Continue.

  7. Ensure that the account information presented is as expected, and if not, amend accordingly.

  8. Complete the following questions where prompted. Include which type of install type you are using, either RPM or embedded-image. Click Continue:

    • What are you experiencing? What are you expecting to happen?

    • Define the value or impact to you or the business.

    • Where are you experiencing this behavior? What environment?

    • When does this behavior occur? Frequency? Repeatedly? At certain times?

  9. Upload relevant diagnostic data files and click Continue. Include data gathered using the sos tool or etcd as a starting point, plus any issue-specific data that is not collected in those logs.

  10. Add relevant case management details and click Continue.

  11. Preview the case details and click Submit.

About sosreport archive

You can use an sosreport archive to troubleshoot a failing host or problems with Red Hat OpenShift GitOps. The sos tool combines plugins that help you gather information from different applications. The sos report command generates a detailed report that shows all of the enabled plugins and data from the different components and applications in a system.

A MicroShift-specific plugin from sos version 4.5.1 gathers the following data:

  • MicroShift configuration and version

  • YAML output for node and system namespaced resources

  • OVN-Kubernetes information

Gather data from an sos report

You can create an sosreport archive about a failing {op-system-full} host that you can share with Red Hat support for troubleshooting.

Prerequisites

  • You must have the sos package installed.

  • You have root access to the host.

Procedure

  1. Log in to the failing host as a root user.

  2. Perform the debug report creation procedure by running the following command:

    $ microshift-sos-report
    Example output
    sosreport (version 4.5.1)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.o0sznf_8 and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

        Distribution Website : https://www.redhat.com/
        Commercial Support   : https://www.access.redhat.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.


 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/2   microshift      [Running: microshift]
  Starting 2/2   microshift_ovn  [Running: microshift microshift_ovn]
  Finishing plugins              [Running: microshift]

  Finished running plugins

Found 1 total reports to obfuscate, processing up to 4 concurrently

sosreport-microshift-rhel9-2023-03-31-axjbyxw :    Beginning obfuscation...
sosreport-microshift-rhel9-2023-03-31-axjbyxw :    Obfuscation completed

Successfully obfuscated 1 report(s)

Creating compressed archive...

A mapping of obfuscated elements is available at
	/var/tmp/sosreport-microshift-rhel9-2023-03-31-axjbyxw-private_map

Your sosreport has been generated and saved in:
	/var/tmp/sosreport-microshift-rhel9-2023-03-31-axjbyxw-obfuscated.tar.xz

 Size	444.14KiB
 Owner	root
 sha256	922e5ff2db25014585b7c6c749d2c44c8492756d619df5e9838ce863f83d4269

Please send this file to your support representative.

Get the node ID of a running node

Retrieving the node ID enables you to uniquely identify a device within your deployment. The node ID is required to register the node with central management systems, analyze system logs and alerts, and ensure that configuration updates are targeted accurately.

Procedure

  • Get the ID of a running node using oc get by entering the following command:

    $ oc get namespaces kube-system -o jsonpath={.metadata.uid}
    Example output
    7cf13853-68f4-454e-8f5c-1af748cbfb1a

  • Get the ID of a running node by retrieving it from the cluster-id file by entering the following command:

    $ sudo cat /var/lib/microshift/cluster-id
    Example output
    7cf13853-68f4-454e-8f5c-1af748cbfb1a

Get the node ID of a stopped node

When the MicroShift service is inactive, you can prevent the use of standard API commands by retrieving the node ID from the file system. You can use this ID to identify offline nodes for disaster recovery, verify backup compatibility, and troubleshoot issues.

Procedure

  • Get the ID of a stopped node by retrieving it from the cluster-id file by entering the following command:

    $ sudo cat /var/lib/microshift/cluster-id
    Example output
    7cf13853-68f4-454e-8f5c-1af748cbfb1a

About remote health monitoring with MicroShift

Remote health monitoring is conducted in MicroShift by the collection of telemetry and configuration data about your node that is reported to Red Hat with the Telemeter API. A node that reports Telemetry to Red Hat is considered a connected node.

Telemetry is the term that Red Hat uses to describe the information being sent to Red Hat by the MicroShift Telemeter API. Lightweight attributes are sent from a connected node to Red Hat to monitor the health of a node.

Telemetry provides the following benefits:

  • Enhanced identification and resolution of issues. Events that might seem normal to an end-user can be observed by Red Hat from a broader perspective. Some issues can be more rapidly identified from this point of view and resolved without an end-user needing to open a support case or file a Jira issue.

  • Targeted prioritization of new features and functionality. The data collected provides information about system capabilities and usage characteristics. With this information, Red Hat can focus on developing the new features and functionality that have the greatest impact for our customers.

Telemetry sends a carefully chosen subset of the node monitoring metrics to Red Hat. The Telemeter API fetches the metrics values every hour and uploads the data to Red Hat. This stream of data is used by Red Hat to monitor nodes over time.

This debugging information is available to Red Hat Support and Engineering teams with the same restrictions as accessing data reported through support cases. All connected node information is used by Red Hat to help make MicroShift better.

Note

MicroShift does not support Prometheus. To view the Telemetry gathered from your node, you must contact Red Hat Support.

Information collected by the MicroShift Telemetry API

The MicroShift Telemetry API collects a lightweight set of metrics to assist with remote health monitoring and product improvement. The data payload is minimal, generally under 2KB, and is designed to have very minimal impact on node resources. The collected information is categorized into system configuration, node capacity, and usage metrics.

The following information is collected by Telemetry:

System information

The system information describes the basic configuration of your MicroShift node and where it is running, for example:

  • Version information, including the MicroShift node version.

  • The Red Hat Enterprise Linux (RHEL) version.

  • The RHEL deployment type.

Sizing information

Sizing information details the node capacity, for example:

  • The CPU cores MicroShift can use.

  • Architecture information.

  • The usable bytes of memory.

Usage information

Usage information outlines what is happening in the node, for example:

  • The CPU usage in percentage.

  • The memory usage in percentage.

  • The number of Kubernetes objects by resource type (CRDs).

  • The number of running containers, namespaces, and running pods.

  • The number of routes, ingress, services.

Note

Telemetry does not collect identifying information such as usernames or passwords. Red Hat does not intend to collect personal information. If Red Hat discovers that personal information has been inadvertently received, Red Hat deletes such information. To the extent that any Telemetry constitutes personal data, refer to the Red Hat Privacy Statement for more information about Red Hat’s privacy practices.
Additional details about how remote health monitoring data is used

Red Hat collects data about your use of the Red Hat product(s) for purposes such as providing support and troubleshooting, improving the offerings and user experience, responding to issues, and for billing purposes if applicable.

  • Collection safeguards: Red Hat employs technical and organizational measures designed to protect Telemetry data.

  • Sharing: Red Hat might share the data collected through the Telemetry API internally within Red Hat to improve your user experience. Red Hat might share Telemetry data with its business partners in an aggregated form that does not identify customers to help the partners better understand their markets and their customers' use of Red Hat offerings, or to ensure the successful integration of products jointly supported by those partners.

  • Third parties: Red Hat might engage certain third parties to assist in the collection, analysis, and storage of Telemetry data.

  • Disabling Telemetry data collection: You can disable MicroShift Telemetry by following the instructions in the "Opting out of remote health reporting for MicroShift" section.

Opt out of Telemetry for MicroShift

By default, MicroShift enables the Telemetry service to collect health and usage data. You can disable this service if your node is operating in a disconnected environment or if you want to opt out of data collection.

Prerequisites

  • You installed OpenShift CLI (oc).

  • You have root access to the node.

Procedure

  1. If you have not done so, make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml.

  2. Keep the new MicroShift config.yaml in the /etc/microshift/ directory. Your config.yaml file is read every time the MicroShift service starts.

    Note

    After you create it, the config.yaml file takes precedence over built-in settings.

  3. Optional: Use a configuration snippet if you are using an existing MicroShift YAML. See "Using configuration snippets" in the Additional resources section for more information.

  4. Set the telemetry.status section of the MicroShift YAML with the Disabled value.

    Example disabled Telemetry configuration
    apiServer:
# ...
telemetry:
    endpoint: https://infogw.api.openshift.com
    status: Disabled
# ...

Use sos reports

The sos tool collects diagnostic information from a MicroShift node, including logs, configuration files, and system state. You can generate an sos report and share it with Red Hat Support to help diagnose problems. MicroShift provides a dedicated sos plugin that gathers MicroShift-specific data alongside standard system information.

About sosreport archive

You can use an sosreport archive to troubleshoot a failing host or problems with Red Hat OpenShift GitOps. The sos tool combines plugins that help you gather information from different applications. The sos report command generates a detailed report that shows all of the enabled plugins and data from the different components and applications in a system.

A MicroShift-specific plugin from sos version 4.5.1 gathers the following data:

  • MicroShift configuration and version

  • YAML output for node and system namespaced resources

  • OVN-Kubernetes information

Gather data from an sos report

You can create an sosreport archive about a failing {op-system-full} host that you can share with Red Hat support for troubleshooting.

Prerequisites

  • You must have the sos package installed.

  • You have root access to the host.

Procedure

  1. Log in to the failing host as a root user.

  2. Perform the debug report creation procedure by running the following command:

    $ microshift-sos-report
    Example output
    sosreport (version 4.5.1)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.o0sznf_8 and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

        Distribution Website : https://www.redhat.com/
        Commercial Support   : https://www.access.redhat.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.


 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/2   microshift      [Running: microshift]
  Starting 2/2   microshift_ovn  [Running: microshift microshift_ovn]
  Finishing plugins              [Running: microshift]

  Finished running plugins

Found 1 total reports to obfuscate, processing up to 4 concurrently

sosreport-microshift-rhel9-2023-03-31-axjbyxw :    Beginning obfuscation...
sosreport-microshift-rhel9-2023-03-31-axjbyxw :    Obfuscation completed

Successfully obfuscated 1 report(s)

Creating compressed archive...

A mapping of obfuscated elements is available at
	/var/tmp/sosreport-microshift-rhel9-2023-03-31-axjbyxw-private_map

Your sosreport has been generated and saved in:
	/var/tmp/sosreport-microshift-rhel9-2023-03-31-axjbyxw-obfuscated.tar.xz

 Size	444.14KiB
 Owner	root
 sha256	922e5ff2db25014585b7c6c749d2c44c8492756d619df5e9838ce863f83d4269

Please send this file to your support representative.

Get your node ID

The MicroShift node ID uniquely identifies your node within the cluster and is used in support cases, licensing, and cluster registration scenarios. You can retrieve the node ID from the kube-system namespace either while MicroShift is running or from a stopped node by reading the stored namespace data.

Get the node ID of a running node

Retrieving the node ID enables you to uniquely identify a device within your deployment. The node ID is required to register the node with central management systems, analyze system logs and alerts, and ensure that configuration updates are targeted accurately.

Procedure

  • Get the ID of a running node using oc get by entering the following command:

    $ oc get namespaces kube-system -o jsonpath={.metadata.uid}
    Example output
    7cf13853-68f4-454e-8f5c-1af748cbfb1a

  • Get the ID of a running node by retrieving it from the cluster-id file by entering the following command:

    $ sudo cat /var/lib/microshift/cluster-id
    Example output
    7cf13853-68f4-454e-8f5c-1af748cbfb1a

Get the node ID of a stopped node

When the MicroShift service is inactive, you can prevent the use of standard API commands by retrieving the node ID from the file system. You can use this ID to identify offline nodes for disaster recovery, verify backup compatibility, and troubleshoot issues.

Procedure

  • Get the ID of a stopped node by retrieving it from the cluster-id file by entering the following command:

    $ sudo cat /var/lib/microshift/cluster-id
    Example output
    7cf13853-68f4-454e-8f5c-1af748cbfb1a

← Previous: Install | ↑ Up: Red Hat build of MicroShift | Next: API →