About short-circuiting machine health check remediation
You can use machine health check short-circuiting to ensure that machine health checks remediate machines only when the cluster is healthy, by configuring the maxUnhealthy field in the MachineHealthCheck resource.
If you define a value for the maxUnhealthy field, before remediating any machines, the MachineHealthCheck compares the value of maxUnhealthy with the number of machines within its target pool that it has determined to be unhealthy. Remediation is not performed if the number of unhealthy machines exceeds the maxUnhealthy limit.
|
|
If |
The appropriate maxUnhealthy value depends on the scale of the cluster you deploy and how many machines the MachineHealthCheck covers. For example, you can use the maxUnhealthy value to cover multiple compute machine sets across multiple availability zones so that if you lose an entire zone, your maxUnhealthy setting prevents further remediation within the cluster. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.
|
|
If you configure a This configuration ensures that the machine health check takes no action when multiple control plane machines appear to be unhealthy. Multiple unhealthy control plane machines can indicate that the etcd cluster is degraded or that a scaling operation to replace a failed machine is in progress. If the etcd cluster is degraded, manual intervention might be required. If a scaling operation is in progress, the machine health check should allow it to finish. |
The maxUnhealthy field can be set as either an integer or percentage.
There are different remediation implementations depending on the maxUnhealthy value.
- Setting maxUnhealthy by using an absolute value
-
If
maxUnhealthyis set to2:-
Remediation will be performed if 2 or fewer nodes are unhealthy
-
Remediation will not be performed if 3 or more nodes are unhealthy
These values are independent of how many machines are being checked by the machine health check.
-
- Setting maxUnhealthy by using percentages
-
If
maxUnhealthyis set to40%and there are 25 machines being checked:-
Remediation will be performed if 10 or fewer nodes are unhealthy
-
Remediation will not be performed if 11 or more nodes are unhealthy
If
maxUnhealthyis set to40%and there are 6 machines being checked:-
Remediation will be performed if 2 or fewer nodes are unhealthy
-
Remediation will not be performed if 3 or more nodes are unhealthy
The allowed number of machines is rounded down when the percentage of
maxUnhealthymachines that are checked is not a whole number. -