Deploying a managed cluster with ClusterInstance and GitOps ZTP

Use the following procedure to create a ClusterInstance custom resource (CR) and related files and initiate the GitOps Zero Touch Provisioning (ZTP) cluster deployment.

Note

You require Red Hat Advanced Cluster Management (RHACM) version 2.12 or later to install the SiteConfig Operator and use the ClusterInstance CR.

Prerequisites
  • You have installed the OpenShift CLI (oc).

  • You installed the SiteConfig Operator in the hub cluster.

  • You have logged in to the hub cluster as a user with cluster-admin privileges.

  • You configured the hub cluster for generating the required installation and policy CRs.

  • You created a Git repository where you manage your custom site configuration data. The repository must be accessible from the hub cluster and you must configure it as a source repository for the ArgoCD application. See "Preparing the GitOps ZTP site configuration repository" for more information.

    Note

    When you create the source repository, ensure that you patch the ArgoCD application with the argocd/deployment/argocd-openshift-gitops-patch.json patch-file that you extract from the ztp-site-generate container. See "Configuring the hub cluster with ArgoCD".

  • To be ready for provisioning managed clusters, you require the following for each bare-metal host:

    Network connectivity

    Your network requires DNS. Managed cluster hosts should be reachable from the hub cluster. Ensure that Layer 3 connectivity exists between the hub cluster and the managed cluster host.

    Baseboard Management Controller (BMC) details

    GitOps ZTP uses BMC username and password details to connect to the BMC during cluster installation. The GitOps ZTP plugin manages the ManagedCluster CRs on the hub cluster based on the ClusterInstance CR in your site Git repo. You create individual BMCSecret CRs for each host manually.

Procedure
  1. Create the required managed cluster secrets on the hub cluster. These resources must be in a namespace with a name matching the cluster name. For example, in out/argocd/example/clusterinstance/example-sno.yaml, the cluster name and namespace is example-sno.

    1. Export the cluster namespace by running the following command:

      $ export CLUSTERNS=example-sno
    2. Create the namespace:

      $ oc create namespace $CLUSTERNS
  2. Create pull secret and BMC Secret CRs for the managed cluster. The pull secret must contain all the credentials necessary for installing OpenShift Container Platform and all required Operators. See "Creating the managed bare-metal host secrets" for more information.

    Note

    The secrets are referenced from the ClusterInstance custom resource (CR) by name. The namespace must match the ClusterInstance namespace.

  3. Create a ClusterInstance CR for your cluster in your local clone of the Git repository:

    1. Choose the appropriate example for your CR from the out/argocd/example/clusterinstance/ folder. The folder includes example files for single node, three-node, and standard clusters:

      • example-sno.yaml

      • example-3node.yaml

      • example-standard.yaml

    2. Change the cluster and host details in the example file to match the type of cluster you want. For example:

      Example single-node OpenShift ClusterInstance CR
      # example-node1-bmh-secret & assisted-deployment-pull-secret need to be created under same namespace example-ai-sno
      ---
      apiVersion: siteconfig.open-cluster-management.io/v1alpha1
      kind: ClusterInstance
      metadata:
        name: "example-ai-sno"
        namespace: "example-ai-sno"
      spec:
        baseDomain: "example.com"
        pullSecretRef:
          name: "assisted-deployment-pull-secret"
        clusterImageSetNameRef: "openshift-4.22"
        sshPublicKey: "ssh-rsa AAAA..."
        clusterName: "example-ai-sno"
        networkType: "OVNKubernetes"
        # installConfigOverrides is a generic way of passing install-config
        # parameters through the siteConfig.  The 'capabilities' field configures
        # the composable openshift feature.  In this 'capabilities' setting, we
        # remove all the optional set of components.
        # Notes:
        # - OperatorLifecycleManager is needed for 4.15 and later
        # - NodeTuning is needed for 4.13 and later, not for 4.12 and earlier
        # - Ingress is needed for 4.16 and later
        installConfigOverrides: |
          {
            "capabilities": {
              "baselineCapabilitySet": "None",
              "additionalEnabledCapabilities": [
                "NodeTuning",
                "OperatorLifecycleManager",
                "Ingress"
              ]
            }
          }
        # Include references to extraManifest ConfigMaps.
        extraManifestsRefs:
          - name: sno-extra-manifest-configmap
        extraLabels:
          ManagedCluster:
            # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples
            du-profile: "latest"
            # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples in ../policygentemplates:
            # ../policygentemplates/common-ranGen.yaml will apply to all clusters with 'common: true'
            common: "true"
            # ../policygentemplates/group-du-sno-ranGen.yaml will apply to all clusters with 'group-du-sno: ""'
            group-du-sno: ""
            # ../policygentemplates/example-sno-site.yaml will apply to all clusters with 'sites: "example-sno"'
            # Normally this should match or contain the cluster name so it only applies to a single cluster
            sites : "example-sno"
        clusterNetwork:
          - cidr: 1001:1::/48
            hostPrefix: 64
        machineNetwork:
          - cidr: 1111:2222:3333:4444::/64
        serviceNetwork:
          - cidr: 1001:2::/112
        additionalNTPSources:
          - 1111:2222:3333:4444::2
        # Initiates the cluster for workload partitioning. Setting specific reserved/isolated CPUSets is done via PolicyTemplate
        # please see Workload Partitioning Feature for a complete guide.
        cpuPartitioningMode: AllNodes
        templateRefs:
          - name: ai-cluster-templates-v1
            namespace: open-cluster-management
        nodes:
          - hostName: "example-node1.example.com"
            role: "master"
            bmcAddress: "idrac-virtualmedia+https://[1111:2222:3333:4444::bbbb:1]/redfish/v1/Systems/System.Embedded.1"
            bmcCredentialsName:
              name: "example-node1-bmh-secret"
            bootMACAddress: "AA:BB:CC:DD:EE:11"
            # Use UEFISecureBoot to enable secure boot, UEFI to disable.
            bootMode: "UEFISecureBoot"
            rootDeviceHints:
              deviceName: "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0"
            # disk partition at `/var/lib/containers` with ignitionConfigOverride. Some values must be updated. See DiskPartitionContainer.md in argocd folder for more details
            ignitionConfigOverride: |
              {
                "ignition": {
                  "version": "3.2.0"
                },
                "storage": {
                  "disks": [
                    {
                      "device": "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0",
                      "partitions": [
                        {
                          "label": "var-lib-containers",
                          "sizeMiB": 0,
                          "startMiB": 250000
                        }
                      ],
                      "wipeTable": false
                    }
                  ],
                  "filesystems": [
                    {
                      "device": "/dev/disk/by-partlabel/var-lib-containers",
                      "format": "xfs",
                      "mountOptions": [
                        "defaults",
                        "prjquota"
                      ],
                      "path": "/var/lib/containers",
                      "wipeFilesystem": true
                    }
                  ]
                },
                "systemd": {
                  "units": [
                    {
                      "contents": "# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target",
                      "enabled": true,
                      "name": "var-lib-containers.mount"
                    }
                  ]
                }
              }
            nodeNetwork:
              interfaces:
                - name: eno1
                  macAddress: "AA:BB:CC:DD:EE:11"
              config:
                interfaces:
                  - name: eno1
                    type: ethernet
                    state: up
                    ipv4:
                      enabled: false
                    ipv6:
                      enabled: true
                      address:
                      # For SNO sites with static IP addresses, the node-specific,
                      # API and Ingress IPs should all be the same and configured on
                      # the interface
                      - ip: 1111:2222:3333:4444::aaaa:1
                        prefix-length: 64
                dns-resolver:
                  config:
                    search:
                    - example.com
                    server:
                    - 1111:2222:3333:4444::2
                routes:
                  config:
                  - destination: ::/0
                    next-hop-interface: eno1
                    next-hop-address: 1111:2222:3333:4444::1
                    table-id: 254
            templateRefs:
              - name: ai-node-templates-v1
                namespace: open-cluster-management
      Note

      For more information about BMC addressing, see the "Additional resources" section. The installConfigOverrides and ignitionConfigOverride fields are expanded in the example for ease of readability.

      Note

      To override the default BareMetalHost CR for a node, create a custom node template in a ConfigMap and reference it in the node-level spec.nodes.templateRefs field in the ClusterInstance CR. Ensure that you set the argocd.argoproj.io/sync-wave: "3" annotation in your override BareMetalHost CR.

    3. You can inspect the default set of extra-manifest MachineConfig CRs in out/argocd/extra-manifest. It is automatically applied to the cluster when it is installed.

    4. Optional: To provision additional install-time manifests on the provisioned cluster, package your extra manifest CRs in a ConfigMap and reference it in the extraManifestsRefs field of the ClusterInstance CR. For more information, see "Customizing extra installation manifests in the GitOps ZTP pipeline".

      Important

      For optimal cluster performance, enable crun for master and worker nodes in single-node OpenShift, single-node OpenShift with additional worker nodes, three-node OpenShift, and standard clusters.

      Enable crun in a ContainerRuntimeConfig CR as an additional Day 0 install-time manifest to avoid the cluster having to reboot.

      The enable-crun-master.yaml and enable-crun-worker.yaml CR files are in the out/source-crs/optional-extra-manifest/ folder that you can extract from the ztp-site-generate container.

  4. Add the ClusterInstance CR to the kustomization.yaml file in the generators section, similar to the example shown in out/argocd/example/clusterinstance/kustomization.yaml.

  5. Commit the ClusterInstance CR and associated kustomization.yaml changes in your Git repository and push the changes.

    The ArgoCD pipeline detects the changes and begins the managed cluster deployment.

Verification
  • Verify that the custom roles and labels are applied after the node is deployed:

    $ oc describe node example-node.example.com

    The following example output shows the custom roles and labels:

    Name:   example-node.example.com
    Roles:  control-plane,example-label,master,worker
    Labels: beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/os=linux
            custom-label/parameter1=true
            kubernetes.io/arch=amd64
            kubernetes.io/hostname=cnfdf03.telco5gran.eng.rdu2.redhat.com
            kubernetes.io/os=linux
            node-role.kubernetes.io/control-plane=
            node-role.kubernetes.io/example-label=
            node-role.kubernetes.io/master=
            node-role.kubernetes.io/worker=
            node.openshift.io/os_id=rhcos
    • node-role.kubernetes.io/example-label= shows the custom label applied to the node.