OpenShift Container Platform

How OpenShift Container Platform enforces system-reserved CPU

By default, OpenShift Container Platform calculates how much CPU should be reserved for system processes and uses cgroup controls to ensure that CPU is distributed according to configured weights and limits.

This enforcement is applied in times of CPU contention within a node. When a node is not under CPU pressure, system processes can use more than the reserved share if the unused CPU is available.

OpenShift Container Platform uses the systemReservedCgroup and enforceNodeAllocatable parameters in the kubelet configuration to perform this CPU enforcement.

By default, the systemReservedCgroup parameter is set to /system.slice, which enforces CPU limits on the systemd system.slice cgroup, which includes kubelet, CRI-O, and other system services. The enforceNodeAllocatable parameter is set to pods and system-reserved-compressible, which enforce node allocatable limits across both pods and the system-reserved resources.

As an example of this enforcement, if 0.5 CPU is reserved for system daemons on a four-core node, the systemd system.slice cgroup receives approximately that much CPU during contention and workload pods receive their expected share of the remaining 3.5 CPUs. If the system daemons hit the CPU limit, the OpenShift Container Platform throttles, but does not kill, the daemons. The OpenShift Container Platform reduces the CPU time allocated to processes in the system.slice cgroup, spreading their work over a longer period.

If other systemd slices are running CPU-intensive workloads, contention from slices other than system.slice could still impact overall CPU allocation.

However, the kubelet cannot simultaneously enforce both the systemReservedCgroup setting and the reservedSystemCPUs setting used by Performance Profiles. If you apply a Performance Profile configured with reservedSystemCPUs, OpenShift Container Platform takes the following actions:

The systemReservedCgroup setting is automatically cleared.
The enforceNodeAllocatable parameter is set to pods only.
The existing Performance Profile behavior is preserved without requiring any changes.

You can disable the default enforcement behavior for nodes in specific machine config pools, if needed, by using a KubeletConfig object. You might disable the feature for the following reasons:

You have specific system daemon resource management requirements.
You want to perform troubleshooting if system daemons are being throttled unexpectedly.

The following kubelet config disables CPU reservation enforcement:

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: 80-custom-system-reserved
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  kubeletConfig:
    apiVersion: kubelet.config.k8s.io/v1beta1
    kind: KubeletConfiguration
    systemReservedCgroup: ""
    enforceNodeAllocatable:
      - "pods"

where:

metadata.name: Specifies a name for the kubelet config. Use a high number prefix to ensure it merges last.
spec.machineConfigPoolSelector.matchLabels: Specifies a label for the machine config pool that you want to modify.
spec.kubeletConfig.systemReservedCgroup: When set to "", specifies that the cgroup enforcement is disabled.
spec.kubeletConfig.enforceNodeAllocatable: When set only to "pods", specifies that node allocatable enforcement applies only across pods and not across the system-reserved resources.