Red Hat OpenShift Container Platform

Health rule threshold and grouping customization

Health rules in the Network Observability Operator are defined by using rule templates and variants in the spec.processor.metrics.healthRules field of the FlowCollector custom resource (CR). Customizing these templates allows for flexible, fine-grained alerting tailored to specific environment needs.

For each template, a list of variants can be defined, each with distinct thresholds and grouping configurations.

The following example shows a FlowCollector configuration with custom health rules:

apiVersion: flows.netobserv.io/v1beta1
kind: FlowCollector
metadata:
  name: flow-collector
spec:
  processor:
    metrics:
      healthRules:
      - template: PacketDropsByKernel
        mode: Alert # or Recording
        variants:
        # Triggered when aggregate cluster traffic reaches 10% drops
        - thresholds:
            critical: "10"
        # Triggered per-node with increasing severity levels
        - thresholds:
            critical: "15"
            warning: "10"
            info: "5"
          groupBy: Node

spec.processor.metrics.healthRules.template: Specifies the name of the predefined rule template.
spec.processor.metrics.healthRules.mode: Specifies whether the rule functions as an Alert or a Recording rule.
spec.processor.metrics.healthRules.variants.thresholds: Specifies the numerical values that trigger the rule. Multiple severity levels, such as critical, warning, or info, can be defined within a single variant.
spec.processor.metrics.healthRules.variants.groupBy: Specifies the dimension used to aggregate the metric, such as Node or Namespace.

Customizing a rule replaces the default configuration for that template. To retain default configurations, the default settings must be manually included in the custom resource.

Health rule query and metadata reference

The FlowCollector health rule API maps to the Prometheus Operator to generate PrometheusRule objects. Use these base Prometheus Query Language (PromQL) patterns and metadata configurations to create custom health rules for network observability.

The PrometheusRule resource in the netobserv namespace can be viewed by running the following command:

$ oc get prometheusrules -n netobserv -o yaml

Customizing alert logic with PromQL: Incoming traffic surge

The following PromQL query calculates the byte rate from the openshift-ingress namespace to any workload namespace over a 30-minute interval:

sum(rate(netobserv_workload_ingress_bytes_total{SrcK8S_Namespace="openshift-ingress"}[30m])) by (DstK8S_Namespace)

Queries can be customized to filter low-bandwidth data, compare time periods, and establish thresholds.

Data filtering

Appending > 1000 to the query removes rates lower than 1 KB/s to filter low-bandwidth traffic.

(sum(rate(netobserv_workload_ingress_bytes_total{SrcK8S_Namespace="openshift-ingress"}[30m])) by (DstK8S_Namespace) > 1000)

The byte rate is relative to the sampling interval in the FlowCollector CR. Normalizing byte rates with the netobserv_agent_sampling_rate metric decouples the PromQL expression from the sampling configuration.

Time comparison

The offset modifier compares data across different time periods. For example, offset 1d retrieves data from the previous day.

sum(rate(netobserv_workload_ingress_bytes_total{SrcK8S_Namespace="openshift-ingress"}[30m] offset 1d)) by (DstK8S_Namespace))

Threshold application

A final threshold filters increases below a specific percentage. For example, > 100 removes increases lower than 100%.

The following example shows a complete PromQL expression for a PrometheusRule:

expr: |-
  (100 *
    (
      (sum(rate(netobserv_workload_ingress_bytes_total{SrcK8S_Namespace="openshift-ingress"}[30m])) by (DstK8S_Namespace) > 1000)
      - sum(rate(netobserv_workload_ingress_bytes_total{SrcK8S_Namespace="openshift-ingress"}[30m] offset 1d)) by (DstK8S_Namespace)
    )
    / sum(rate(netobserv_workload_ingress_bytes_total{SrcK8S_Namespace="openshift-ingress"}[30m] offset 1d)) by (DstK8S_Namespace))
  > 100

Alert metadata fields

Rule definitions require specific metadata for the Prometheus Alertmanager service and the Network Health dashboard. The following example shows an AlertingRule resource with configured metadata:

apiVersion: monitoring.openshift.io/v1
kind: AlertingRule
metadata:
  name: netobserv-alerts
  namespace: openshift-monitoring
spec:
  groups:
  - name: NetObservAlerts
    rules:
    - alert: NetObservIncomingBandwidth
      annotations:
        netobserv_io_network_health: '{"namespaceLabels":["DstK8S_Namespace"],"threshold":"100","unit":"%","upperBound":"500"}'
        message: |-
          Surge of incoming traffic detected: current traffic to {{ $labels.DstK8S_Namespace }} increased by more than 100% since yesterday.
        summary: "Surge in incoming traffic"
      expr: |-
        # ... (PromQL expression)
      for: 1m
      labels:
        app: netobserv
        netobserv: "true"
        severity: warning

spec.groups.rules.alert.labels.netobserv: Specifies that the Network Health dashboard must detect the alert when set to true.
spec.groups.rules.alert.labels.severity: Specifies the alert severity. Valid values are critical, warning, or info.

netobserv_io_network_health annotation fields

The optional netobserv_io_network_health annotation is a JSON string that controls how the alert renders on the Network Health page.

Table 17. Fields for the `netobserv_io_network_health` annotation
Field	Type	Description
`namespaceLabels`	List of strings	One or more labels containing namespaces. Alerts appear under the Namespaces tab.
`nodeLabels`	List of strings	One or more labels containing node names. Alerts appear under the Nodes tab.
`workloadLabels`	List of strings	One or more labels containing owner or workload names. Alerts appear under the Owners tab when `kindLabels` is also provided.
`threshold`	String	The alert threshold. This value should match the threshold in the PromQL expression.
`unit`	String	The data unit for display purposes.
`upperBound`	String	An upper bound value used to calculate scores on a closed scale. Metric values exceeding this bound are clamped.

The namespaceLabels and nodeLabels fields are mutually exclusive. If neither is provided, the alert appears under the Global tab.

Configuring custom health rules

Create custom health rules by using Prometheus Query Language (PromQL) to define an AlertingRule resource. These rules trigger alerts based on specific network metrics, such as traffic surges.

Prerequisites

Access to the cluster with cluster-admin privileges.
The Network Observability Operator is installed.
Red Hat OpenShift Container Platform 4.16 or later is installed.
Familiarity with PromQL.

Custom PrometheusRule resources are not owned by the FlowCollector resource. Custom rules created in the netobserv namespace might be deleted if the Network Observability Operator is uninstalled. To prevent data loss, create custom rules in a different namespace, such as openshift-monitoring, and maintain a backup in version control.

Procedure

Define an AlertingRule resource in a YAML file, for example, custom-alert.yaml.
Apply the custom alert rule by running the following command:
```
$ oc apply -f custom-alert.yaml
```

Verification

Confirm the PrometheusRule resource was created in the target namespace by running the following command:
```
$ oc get prometheusrules -n <namespace> -o yaml
```
Confirm the rule is active in the Red Hat OpenShift Container Platform web console:
1. Navigate to Observe → Alerting to see the firing status.
2. Navigate to Observe → Network Health to view the dashboard integration.