Test a workload health check script

You can look at the output of the greenboot workload health check script to see how it works. While the output varies with the host system type, the example outputs for Red Hat Enterprise Linux (RHEL) system types are included for reference only.

Prerequisites

  • You have root access.

  • You installed a workload.

  • You created a health check script for the workload.

  • The MicroShift service is enabled.

Procedure

  1. To test that greenboot is running a health check script file, reboot the host by running the following command:

    $ sudo reboot

  2. Examine the output of greenboot health checks by running the following command:

    $ sudo journalctl -o cat -u greenboot-healthcheck.service
    Note

    MicroShift core service health checks run before the workload health checks.
    Example output for an image mode for RHEL system
    Starting greenboot Health Checks Runner...
Running Required Health Check Scripts...
Script '00_required_scripts_start.sh' SUCCESS
Running Wanted Health Check Scripts...
Script '00_wanted_scripts_start.sh' SUCCESS
Running Required Health Check Scripts...
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 11:54:30.526488     979 service.go:29] microshift.service is enabled
??? I0403 11:54:30.527145     979 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 11:58:52.530299     979 service.go:38] microshift.service is ready
??? I0403 11:58:52.532292     979 net.go:79] host gateway IP address: 192.168.112.125
??? I0403 11:58:52.555077     979 microshift_core_workloads.go:71] vgs reported: {"report":[{"vg":[{"vg_name":"rhel"}]}],"log":[]}
??? I0403 11:58:52.555138     979 microshift_core_workloads.go:93] Detected 1 volume group (rhel) - LVMS is expected
??? I0403 11:58:52.555143     979 microshift_core_workloads.go:126] Configured optional CSI components: []
??? I0403 11:58:52.555147     979 microshift_core_workloads.go:117] At least one CSI Component is enabled
??? I0403 11:58:52.555770     979 utils.go:34] Waiting for 9 goroutines
??? I0403 11:58:52.555791     979 workloads.go:94] Waiting 10m0s for deployment/service-ca in openshift-service-ca
??? I0403 11:58:52.555890     979 workloads.go:58] Waiting 10m0s for daemonset/ovnkube-master in openshift-ovn-kubernetes
??? I0403 11:58:52.555999     979 workloads.go:94] Waiting 10m0s for deployment/router-default in openshift-ingress
??? I0403 11:58:52.556096     979 workloads.go:58] Waiting 10m0s for daemonset/dns-default in openshift-dns
??? I0403 11:58:52.556244     979 workloads.go:58] Waiting 10m0s for daemonset/ovnkube-node in openshift-ovn-kubernetes
??? I0403 11:58:52.556330     979 workloads.go:94] Waiting 10m0s for deployment/lvms-operator in openshift-storage
??? I0403 11:58:52.556382     979 workloads.go:58] Waiting 10m0s for daemonset/vg-manager in openshift-storage
??? I0403 11:58:52.556425     979 workloads.go:94] Waiting 10m0s for deployment/csi-snapshot-controller in kube-system
??? I0403 11:58:52.556474     979 workloads.go:58] Waiting 10m0s for daemonset/node-resolver in openshift-dns
??? I0403 11:58:52.574284     979 workloads.go:89] Daemonset/ovnkube-node in openshift-ovn-kubernetes is ready
??? I0403 11:58:52.574344     979 workloads.go:89] Daemonset/dns-default in openshift-dns is ready
??? I0403 11:59:12.871058     979 workloads.go:89] Daemonset/node-resolver in openshift-dns is ready
??? I0403 11:59:12.871621     979 workloads.go:89] Daemonset/ovnkube-master in openshift-ovn-kubernetes is ready
??? I0403 11:59:12.871748     979 workloads.go:132] Deployment/csi-snapshot-controller in kube-system is ready
??? I0403 11:59:25.175015     979 workloads.go:132] Deployment/service-ca in openshift-service-ca is ready
??? I0403 11:59:42.559264     979 workloads.go:132] Deployment/lvms-operator in openshift-storage is ready
??? I0403 11:59:52.557786     979 workloads.go:132] Deployment/router-default in openshift-ingress is ready
??? I0403 11:59:52.558489     979 workloads.go:89] Daemonset/vg-manager in openshift-storage is ready
??? I0403 11:59:52.558505     979 healthcheck.go:28] MicroShift is ready
Script '40_microshift_running_check.sh' SUCCESS
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 11:59:52.750474    4059 service.go:29] microshift.service is enabled
??? I0403 11:59:52.750873    4059 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 11:59:52.752273    4059 service.go:38] microshift.service is ready
??? I0403 11:59:52.753263    4059 utils.go:34] Waiting for 1 goroutines
??? I0403 11:59:52.753393    4059 workloads.go:94] Waiting 10m0s for deployment/kserve-controller-manager in redhat-ods-applications
??? I0403 12:00:02.755475    4059 workloads.go:132] Deployment/kserve-controller-manager in redhat-ods-applications is ready
??? I0403 12:00:02.755605    4059 healthcheck.go:75] Workloads are ready
Script '41_microshift_running_check_ai_model_serving.sh' SUCCESS
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 12:00:02.896949    4128 service.go:29] microshift.service is enabled
??? I0403 12:00:02.897208    4128 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 12:00:02.899492    4128 service.go:38] microshift.service is ready
??? I0403 12:00:02.900279    4128 utils.go:34] Waiting for 2 goroutines
??? I0403 12:00:02.900363    4128 workloads.go:94] Waiting 10m0s for deployment/istiod-openshift-gateway-api in openshift-gateway-api
??? I0403 12:00:02.900948    4128 workloads.go:94] Waiting 10m0s for deployment/servicemesh-operator3 in openshift-gateway-api
??? I0403 12:00:42.913338    4128 workloads.go:132] Deployment/servicemesh-operator3 in openshift-gateway-api is ready
??? I0403 12:01:12.902297    4128 workloads.go:132] Deployment/istiod-openshift-gateway-api in openshift-gateway-api is ready
??? I0403 12:01:12.902418    4128 healthcheck.go:75] Workloads are ready
Script '41_microshift_running_check_gateway_api.sh' SUCCESS
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 12:01:13.057998    4772 service.go:29] microshift.service is enabled
??? I0403 12:01:13.058107    4772 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 12:01:13.059839    4772 service.go:38] microshift.service is ready
??? I0403 12:01:13.060617    4772 utils.go:34] Waiting for 2 goroutines
??? I0403 12:01:13.060644    4772 workloads.go:58] Waiting 10m0s for daemonset/dhcp-daemon in openshift-multus
??? I0403 12:01:13.060686    4772 workloads.go:58] Waiting 10m0s for daemonset/multus in openshift-multus
??? I0403 12:01:13.069341    4772 workloads.go:89] Daemonset/multus in openshift-multus is ready
??? I0403 12:01:13.069450    4772 workloads.go:89] Daemonset/dhcp-daemon in openshift-multus is ready
??? I0403 12:01:13.069503    4772 healthcheck.go:75] Workloads are ready
Script '41_microshift_running_check_multus.sh' SUCCESS
--------------------
DEPRECATION NOTICE:
/usr/share/microshift/functions/greenboot.sh is now deprecated and will be removed in future release.
Planned removal: MicroShift 4.21
As a replacement consider using 'microshift healthcheck' command
--------------------
STARTED
GRUB boot variables:
boot_success=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_GREENBOOT_FAIL_MARKER=/run/microshift-greenboot-healthcheck-failed
System installation type:
bootc
System installation status:
bootcHost
??? I0403 12:01:13.206381    4804 service.go:29] microshift.service is enabled
??? I0403 12:01:13.206583    4804 service.go:31] Waiting 10m0s for microshift.service to be ready
??? I0403 12:01:13.207979    4804 service.go:38] microshift.service is ready
??? I0403 12:01:13.208717    4804 utils.go:34] Waiting for 2 goroutines
??? I0403 12:01:13.208779    4804 workloads.go:94] Waiting 10m0s for deployment/catalog-operator in openshift-operator-lifecycle-manager
??? I0403 12:01:13.209285    4804 workloads.go:94] Waiting 10m0s for deployment/olm-operator in openshift-operator-lifecycle-manager
??? I0403 12:01:13.215578    4804 workloads.go:132] Deployment/catalog-operator in openshift-operator-lifecycle-manager is ready
??? I0403 12:01:13.215673    4804 workloads.go:132] Deployment/olm-operator in openshift-operator-lifecycle-manager is ready
??? I0403 12:01:13.215684    4804 healthcheck.go:75] Workloads are ready
Script '50_microshift_running_check_olm.sh' SUCCESS
Running Wanted Health Check Scripts...
Finished greenboot Health Checks Runner.
    Example partial output for a RHEL for Edge system
    #...
GRUB boot variables:
boot_success=0
boot_indeterminate=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
MICROSHIFT_WAIT_TIMEOUT_SEC=600
System installation type:
ostree
System installation status:
* rhel 19619bd269094510180c845c44d0944fd9aa15925376f249c4d680a3355e51ae.0
    Version: 9.6
    origin refspec: edge:rhel-9.6-microshift-4.20
#...
    Example partial output for an RPM system
    #...
GRUB boot variables:
boot_success=1
boot_indeterminate=0
Greenboot variables:
GREENBOOT_WATCHDOG_CHECK_ENABLED=true
System installation type:
RPM
System installation status:
Not an ostree / bootc system
#...

← Previous: Example and functional requirements | ↑ Up: Maintain system health with the greenboot health check framework | ⌂ Home: Red Hat build of MicroShift | Next: Check the status of greenboot health checks →