Red Hat build of MicroShift

Use automatic recovery in RPM systems

To use automatic recovery for {product-title} on RPM systems, you can create the 10-auto-recovery.conf file, the microshift-auto-recovery.service unit, and the microshift-auto-recovery script. Systemd runs the recovery service when the {product-title} service does not start, and the script restores the latest backup.

As a use case, consider the following example situation in which you want to automate the automatic recovery process for RPM systems that use the systemd service.

Procedure

Create a directory for the microshift systemd service by running the following command:
```
$ sudo mkdir -p /usr/lib/systemd/system/microshift.service.d
```
To instruct systemd to run microshift-auto-recovery.service when the microshift.service fails, create the 10-auto-recovery.conf file by running the following command:
```
$ sudo tee /usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf > /dev/null <<'EOF'
[Unit]
OnFailure=microshift-auto-recovery.service
StartLimitIntervalSec=25s

[Service]
RestartMode=direct
EOF
```
- For StartLimitIntervalSec, specify a value greater than the default 10s for slower systems. A value that is too low can result in systemd never marking the microshift systemd service as failed, which means that the OnFailure= service does not get triggered.
- RestartMode=direct prevents systemd from entering failed state on every restart attempt. This ensures OnFailure is triggered only after StartLimitBurst is exceeded, not on each failure. In systemd v254 (RHEL-10), OnFailure behavior changed to trigger on every failure instead of only when restart limits are reached. RestartMode=direct restores the v249 behavior. This setting is ignored on RHEL-9.6 (systemd v252) where it does not exist.

Create the microshift-auto-recovery.service file by running the following command:

$ sudo tee /usr/lib/systemd/system/microshift-auto-recovery.service > /dev/null <<'EOF'
[Unit]
Description=MicroShift auto-recovery

[Service]
Type=oneshot
ExecStart=/usr/bin/microshift-auto-recovery

[Install]
WantedBy=multi-user.target
EOF

Create the microshift-auto-recovery script by running the following command:

$ sudo tee /usr/bin/microshift-auto-recovery > /dev/null <<'EOF'
#!/usr/bin/env bash
set -xeuo pipefail

# If greenboot uses a non-default file for clearing boot_counter, use boot_success instead.
if grep -q  "/boot/grubenv" /usr/libexec/greenboot/greenboot-grub2-set-success; then
    if grub2-editenv - list | grep -q ^boot_success=0; then
        echo "Greenboot didn't decide the system is healthy after staging new deployment."
        echo "Quitting to not interfere with the process"
        exit 0
    fi
else
    if grub2-editenv - list | grep -q ^boot_counter=; then
        echo "Greenboot didn't decide the system is healthy after staging a new deployment."
        echo "Quitting to not interfere with the process"
        exit 0
    fi
fi

/usr/bin/microshift restore --auto-recovery /var/lib/microshift-auto-recovery
/usr/bin/systemctl reset-failed microshift
/usr/bin/systemctl start microshift

echo "DONE"
EOF

Make the script executable by running the following command:
```
$ sudo chmod +x /usr/bin/microshift-auto-recovery
```
Reload the system configuration by running the following command:
```
$ sudo systemctl daemon-reload
```