Red Hat OpenShift Container Platform

About the ClusterUserDefinedNetwork CR

The ClusterUserDefinedNetwork (CUDN) custom resource (CR) provides cluster-scoped network segmentation in Red Hat OpenShift Container Platform and isolation for administrators only. Defining this resource ensures that network traffic is securely partitioned across the entire cluster.

The following diagram demonstrates how a cluster administrator can use the CUDN CR to create network isolation between tenants. This network configuration allows a network to span across many namespaces. In the diagram, network isolation is achieved through the creation of two user-defined networks, udn-1 and udn-2. These networks are not connected and the spec.namespaceSelector.matchLabels field is used to select different namespaces. For example, udn-1 configures and isolates communication for namespace-1 and namespace-2, while udn-2 configures and isolates communication for namespace-3 and namespace-4. Isolated tenants (Tenants 1 and Tenants 2) are created by separating namespaces while also allowing pods in the same namespace to communicate.

The tenant isolation concept in a user-defined network (UDN)

Figure 19. Tenant isolation using a ClusterUserDefinedNetwork CR

Considerations for ClusterUserDefinedNetwork transport

Unlike the UserDefinedNetwork (UDN) custom resource (CR), the ClusterUserDefinedNetwork CR gives you more control over how pod traffic is carried on the cluster infrastructure and how it relates to networks outside the cluster.

By default, pod-to-pod traffic on the CUDN CR uses a Geneve overlay. Pod IP addresses are not directly reachable from outside of the cluster. When workload traffic leaves the cluster through the designated egress gateway, source addresses are masqueraded to the node IP address of the node that forwards the traffic, similar to other pod networks.

You can use route advertisements and the RouteAdvertisements CR so that routes for the CUDN are advertised on the provider network by using Border Gateway Protocol (BGP). Collectively, this configuration makes pod IP addresses reachable from outside the cluster. For information, see "About route advertisements".

Additionally, you can set spec.network.transport to NoOverlay to route layer 3 pod traffic on the underlay with BGP instead of Geneve encapsulation, or to EVPN to attach a primary CUDN to an external BGP EVPN fabric instead of using only the default overlay behavior. Configuring either transport requires additional objects and node networking beyond the CUDN CR. For more information, see "Improve east-west performance by routing pods on the underlay with BGP" and "About BGP EVPN for primary cluster user-defined networks".

Best practices for ClusterUserDefinedNetwork CRs

To create and deploy a successful instance of the ClusterUserDefinedNetwork (CUDN) CR, administrators must follow best practices such as avoiding default and openshift-* namespaces, use the proper namespace selector configuration, and ensure physical network parameter matching.

The following details provide administrators with a best practice for designing a CUDN CR:

A ClusterUserDefinedNetwork CR is intended for use by cluster administrators and should not be used by non-administrators. If used incorrectly, it might result in security issues with your deployment, cause disruptions, or break the cluster network.
ClusterUserDefinedNetwork CRs should not select the default namespace. This can result in no isolation and, as a result, could introduce security risks to the cluster.
ClusterUserDefinedNetwork CRs should not select openshift-* namespaces.
Red Hat OpenShift Container Platform administrators should be aware that all namespaces of a cluster are selected when one of the following conditions are met:
- The matchLabels selector is left empty.
- The matchExpressions selector is left empty.
- The namespaceSelector is initialized, but does not specify matchExpressions or matchLabel. For example: namespaceSelector: {}.
For primary networks, the namespace used for the ClusterUserDefinedNetwork CR must include the k8s.ovn.org/primary-user-defined-network label. This label cannot be updated, and can only be added when the namespace is created. The following conditions apply with the k8s.ovn.org/primary-user-defined-network namespace label:
- If the namespace is missing the k8s.ovn.org/primary-user-defined-network label and a pod is created, the pod attaches itself to the default network.
- If the namespace is missing the k8s.ovn.org/primary-user-defined-network label and a primary ClusterUserDefinedNetwork CR is created that matches the namespace, an error is reported and the network is not created.
- If the namespace is missing the k8s.ovn.org/primary-user-defined-network label and a primary ClusterUserDefinedNetwork CR already exists, a pod in the namespace is created and attached to the default network.
- If the namespace has the label, and a primary ClusterUserDefinedNetwork CR does not exist, a pod in the namespace is not created until the ClusterUserDefinedNetwork CR is created.
When using the ClusterUserDefinedNetwork CR to create localnet topology, the following are best practices for administrators:
- You must make sure that the spec.network.physicalNetworkName parameter matches the parameter that you configured in the Open vSwitch (OVS) bridge mapping when you create your CUDN CR. This ensures that you are bridging to the intended segment of your physical network. If you intend to deploy multiple CUDN CR using the same bridge mapping, you must ensure that the same physicalNetworkName parameter is used.
- Avoid overlapping subnets between your physical network and your other network interfaces. Overlapping network subnets can cause routing conflicts and network instability. To prevent conflicts when using the spec.network.localnet.subnets parameter, you might use the spec.network.localnet.excludeSubnets parameter.
- When you configure a Virtual Local Area Network (VLAN), you must ensure that both your underlying physical infrastructure (switches, routers, and so on) and your nodes are properly configured to accept VLAN IDs (VIDs). This means that you configure the physical network interface, for example eth1, as an access port for the VLAN, for example 20, that you are connecting to through the physical switch. In addition, you must verify that an OVS bridge mapping, for example eth1, exists on your nodes to ensure that the physical interface is properly connected with OVN-Kubernetes.

Creating a ClusterUserDefinedNetwork CR by using the CLI

To implement cluster-wide network segmentation and isolation across multiple namespaces, supporting either layer 2 or layer 3 in Red Hat OpenShift Container Platform, create a ClusterUserDefinedNetwork CR by using the CLI. Defining this resource ensures that network traffic is securely partitioned across the cluster.

Based upon your use case, create your request by using either the cluster-layer-two-udn.yaml example for a Layer2 topology type or the cluster-layer-three-udn.yaml example for a Layer3 topology type.

The ClusterUserDefinedNetwork CR is intended for use by cluster administrators and should not be used by non-administrators. If used incorrectly, it might result in security issues with your deployment, cause disruptions, or break the cluster network.
OpenShift Virtualization only supports the Layer2 and Localnet topologies.

Prerequisites

You have logged in as a user with cluster-admin privileges.

Procedure

Optional: For a ClusterUserDefinedNetwork CR that uses a primary network, create a namespace with the k8s.ovn.org/primary-user-defined-network label by entering the following command:
```
$ cat << EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: <cudn_namespace_name>
  labels:
    k8s.ovn.org/primary-user-defined-network: ""
EOF
```
Create a cluster-wide user-defined network for either a Layer2 or Layer3 topology type:
1. Create a YAML file, such as cluster-layer-two-udn.yaml, to define your request for a Layer2 topology as in the following example:
  apiVersion: k8s.ovn.org/v1 kind: ClusterUserDefinedNetwork metadata: name: <cudn_name> spec: namespaceSelector: matchLabels: "<label_1_key>": "<label_1_value>" "<label_2_key>": "<label_2_value>" network: topology: Layer2 layer2: role: Primary subnets: - "2001:db8::/64" - "10.100.0.0/16" transport: <transport_protocol>
  where:
  Name
  
  Specifies the name of your ClusterUserDefinedNetwork CR.
  
  namespaceSelector
  
  Specifies a label query over the set of namespaces that the CUDN CR applies to. Uses the standard Kubernetes MatchLabel selector. Must not point to default or openshift-* namespaces.
  
  matchLabels
  
  Uses the matchLabels selector type, where terms are evaluated with an AND relationship. In this example, the CUDN CR is deployed to namespaces that contain both <label_1_key>=<label_1_value> and <label_2_key>=<label_2_value> labels.
  
  network
  
  Describes the network configuration.
  
  topology
  
  This field describes the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer2 topology type creates one logical switch that is shared by all nodes. This field specifies the topology configuration. It can be layer2 or layer3.
  
  role
  
  Specifies Primary or Secondary. Primary is the only role specification supported in {product-version}.
  
  subnets
  
  For Layer2 topology types the following specifies config details for the field:
  
  The subnets field is optional.
  
  The subnets field is of type string and accepts standard CIDR formats for both IPv4 and IPv6.
  
  The subnets field accepts one or two items. For two items, they must be of a different family. For example, subnets values of 10.100.0.0/16 and 2001:db8::/64.
  
  Layer2 subnets can be omitted. If omitted, users must configure static IP addresses for the pods. As a consequence, port security only prevents MAC spoofing. For more information, see "Configuring pods with a static IP address".
  
  spec.network.transport
  
  Specifies how pod traffic is carried on the cluster infrastructure for the ClusterUserDefinedNetwork CR. Accepted value is EVPN. Additional configuration is required when setting the spec.network.transport field. This field is optional. For more information, see "About BGP EVPN for primary cluster user-defined networks".
2. Create a YAML file, such as cluster-layer-three-udn.yaml, to define your request for a Layer3 topology as in the following example:
  apiVersion: k8s.ovn.org/v1 kind: ClusterUserDefinedNetwork metadata: name: <cudn_name> spec: namespaceSelector: matchExpressions: - key: kubernetes.io/metadata.name operator: In values: ["<example_namespace_one>", "<example_namespace_two>"] network: topology: Layer3 layer3: role: Primary subnets: - cidr: 10.100.0.0/16 hostSubnet: 24 transport: <transport_protocol>
  where:
  Name
  
  Specifies the name of your ClusterUserDefinedNetwork CR.
  
  namespaceSelector
  
  Specifies a label query over the set of namespaces that the CUDN CR applies to. Uses the standard Kubernetes MatchLabel selector. Must not point to default or openshift-* namespaces. Uses the matchExpressions selector type, where terms are evaluated with an OR relationship.
  
  Key
  
  Specifies the label key to match. Takes an operator value; valid values include: In, NotIn, Exists, and DoesNotExist. Because the matchExpressions type is used, provisions namespaces matching either <example_namespace_one> or <example_namespace_two>.
  
  network
  
  Describes the network configuration.
  
  topology
  
  The topology field describes the network configuration; accepted values are Layer2 and Layer3. Specifying a Layer3 topology type creates a layer 2 segment per node, each with a different subnet. Layer 3 routing is used to interconnect node subnets.
  
  role
  
  Specifies Primary or Secondary. Primary is the only role specification supported in {product-version}.
  
  subnets
  
  For Layer3 topology types the following specifies config details for the subnet field:
  
  The subnets field is mandatory.
  
  The type for the subnets field is cidr and hostSubnet:
  
  cidr is the cluster subnet and accepts a string value.
  
  hostSubnet specifies the nodes subnet prefix that the cluster subnet is split to.
  
  For IPv6, only a /64 length is supported for hostSubnet.
  
  spec.network.transport
  
  Specifies how pod traffic is carried on the cluster infrastructure for the ClusterUserDefinedNetwork CR. Accepted value is EVPN. Additional configuration is required when setting the spec.network.transport field. This field is optional. For more information, see "About BGP EVPN for primary cluster user-defined networks".
Apply your request by running the following command:
```
$ oc create --validate=true -f <example_cluster_udn>.yaml
```
Where <example_cluster_udn>.yaml is the name of your Layer2 or Layer3 configuration file.

Verify that your request is successful by running the following command:

$ oc get clusteruserdefinednetwork <cudn_name> -o yaml

Where <cudn_name> is the name you created of your cluster-wide user-defined network.

Example output

apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  creationTimestamp: "2024-12-05T15:53:00Z"
  finalizers:
  - k8s.ovn.org/user-defined-network-protection
  generation: 1
  name: my-cudn
  resourceVersion: "47985"
  uid: 16ee0fcf-74d1-4826-a6b7-25c737c1a634
spec:
  namespaceSelector:
    matchExpressions:
    - key: custom.network.selector
      operator: In
      values:
      - example-namespace-1
      - example-namespace-2
      - example-namespace-3
  network:
    layer3:
      role: Primary
      subnets:
      - cidr: 10.100.0.0/16
    topology: Layer3
status:
  conditions:
  - lastTransitionTime: "2024-11-19T16:46:34Z"
    message: 'NetworkAttachmentDefinition has been created in following namespaces:
      [example-namespace-1, example-namespace-2, example-namespace-3]'
    reason: NetworkAttachmentDefinitionReady
    status: "True"
    type: NetworkCreated

Creating a ClusterUserDefinedNetwork CR for a Localnet topology

You deploy a Localnet topology to connect the secondary network to the physical underlay. This enables both east-west cluster traffic and access to services running outside the cluster. This topology type requires the additional configuration of the underlying Open vSwitch (OVS) system on cluster nodes.

Prerequisites

You are logged in as a user with cluster-admin privileges.
You created and configured the Open vSwitch (OVS) bridge mapping to associate the logical OVN-Kubernetes network with the physical node network through the OVS bridge. For more information, see "Configuration for a localnet switched topology".

Procedure

Create a cluster-wide user-defined network with a Localnet topology:
1. Create a YAML file, such as cluster-udn-localnet.yaml, to define your request for a Localnet topology as in the following example:
  apiVersion: k8s.ovn.org/v1 kind: ClusterUserDefinedNetwork metadata: name: <cudn_name> spec: namespaceSelector: matchLabels: "<label_1_key>": "<label_1_value>" "<label_2_key>": "<label_2_value>" network: topology: Localnet localnet: role: Secondary physicalNetworkName: test ipam: {lifecycle: Persistent} subnets: ["192.168.0.0/16", "2001:dbb::/64"]
  where:
  Name
  
  Specifies the name of your ClusterUserDefinedNetwork CR.
  
  namespaceSelector
  
  Specifies a label query over the set of namespaces that the CUDN CR applies to. Uses the standard Kubernetes MatchLabel selector. Must not point to default or openshift-* namespaces.
  
  matchLabels
  
  Uses the matchLabels selector type, where terms are evaluated with an AND relationship. In this example, the CUDN CR is deployed to namespaces that contain both <label_1_key>=<alabel_1_value> and <label_2_key>=<label_2_value> labels.
  
  network
  
  Describes the network configuration.
  
  topology
  
  Specifying a Localnet topology type creates one logical switch that is directly bridged to one provider network.
  
  role
  
  Specifies the role for the network configuration. Secondary is the only role specification supported for the localnet topology.
  
  subnets
  
  For Localnet topology types the following specifies config details for the subnet field:
  
  The subnets field is optional.
  
  The subnets field is of type string and accepts standard CIDR formats for both IPv4 and IPv6.
  
  The subnets field accepts one or two items. For two items, they must be of a different IP family. For example, subnets values of 10.100.0.0/16 and 2001:db8::/64.
  
  localnet subnets can be omitted. If omitted, users must configure static IP addresses for the pods. As a consequence, port security only prevents MAC spoofing. For more information, see "Configuring pods with a static IP address".
Apply your request by running the following command:
```
$ oc create --validate=true -f <example_cluster_udn>.yaml
```
where:

<example_cluster_udn>.yaml

Is the name of your Localnet configuration file.
Verify that your request is successful by running the following command:
```
$ oc get clusteruserdefinednetwork <cudn_name> -o yaml
```
where:

<cudn_name>

Is the name you created of your cluster-wide user-defined network.

Example output

apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  creationTimestamp: "2025-05-28T19:30:38Z"
  finalizers:
  - k8s.ovn.org/user-defined-network-protection
  generation: 1
  name: cudn-test
  resourceVersion: "140936"
  uid: 7ff185fa-d852-4196-858a-8903b58f6890
spec:
  namespaceSelector:
    matchLabels:
      "1": "1"
      "2": "2"
  network:
    localnet:
      ipam:
        lifecycle: Persistent
      physicalNetworkName: test
      role: Secondary
      subnets:
      - 192.168.0.0/16
      - 2001:dbb::/64
    topology: Localnet
status:
  conditions:
  - lastTransitionTime: "2025-05-28T19:30:38Z"
    message: 'NetworkAttachmentDefinition has been created in following namespaces:
      [test1, test2]'
    reason: NetworkAttachmentDefinitionCreated
    status: "True"
    type: NetworkCreated

Creating a ClusterUserDefinedNetwork CR by using the web console

To implement isolated network segments with layer 2 connectivity in Red Hat OpenShift Container Platform, create a ClusterUserDefinedNetwork custom resource (CR) by using the web console. Defining this resource ensures that your cluster workloads can communicate directly at the data link layer.

Currently, creation of a ClusterUserDefinedNetwork CR with a Layer3 topology is not supported when using the Red Hat OpenShift Container Platform web console.

Prerequisites

You have access to the Red Hat OpenShift Container Platform web console as a user with cluster-admin permissions.
You have created a namespace and applied the k8s.ovn.org/primary-user-defined-network label.

Procedure

From the Administrator perspective, click Networking → UserDefinedNetworks.
Click ClusterUserDefinedNetwork.
In the Name field, specify a name for the cluster-scoped UDN.
Specify a value in the Subnet field.
In the Project(s) Match Labels field, add the appropriate labels to select namespaces that the cluster UDN applies to.
Click Create. The cluster-scoped UDN serves as the default primary network for pods located in namespaces that contain the labels that you specified in step 5.