Deploying a managed cluster with ClusterInstance and GitOps ZTP
Use the following procedure to create a ClusterInstance custom resource (CR) and related files and initiate the GitOps Zero Touch Provisioning (ZTP) cluster deployment.
|
|
You require Red Hat Advanced Cluster Management (RHACM) version 2.12 or later to install the SiteConfig Operator and use the |
-
You have installed the OpenShift CLI (
oc). -
You installed the SiteConfig Operator in the hub cluster.
-
You have logged in to the hub cluster as a user with
cluster-adminprivileges. -
You configured the hub cluster for generating the required installation and policy CRs.
-
You created a Git repository where you manage your custom site configuration data. The repository must be accessible from the hub cluster and you must configure it as a source repository for the ArgoCD application. See "Preparing the GitOps ZTP site configuration repository" for more information.
When you create the source repository, ensure that you patch the ArgoCD application with the
argocd/deployment/argocd-openshift-gitops-patch.jsonpatch-file that you extract from theztp-site-generatecontainer. See "Configuring the hub cluster with ArgoCD". -
To be ready for provisioning managed clusters, you require the following for each bare-metal host:
- Network connectivity
-
Your network requires DNS. Managed cluster hosts should be reachable from the hub cluster. Ensure that Layer 3 connectivity exists between the hub cluster and the managed cluster host.
- Baseboard Management Controller (BMC) details
-
GitOps ZTP uses BMC username and password details to connect to the BMC during cluster installation. The GitOps ZTP plugin manages the
ManagedClusterCRs on the hub cluster based on theClusterInstanceCR in your site Git repo. You create individualBMCSecretCRs for each host manually.
-
Create the required managed cluster secrets on the hub cluster. These resources must be in a namespace with a name matching the cluster name. For example, in
out/argocd/example/clusterinstance/example-sno.yaml, the cluster name and namespace isexample-sno.-
Export the cluster namespace by running the following command:
$ export CLUSTERNS=example-sno -
Create the namespace:
$ oc create namespace $CLUSTERNS
-
-
Create pull secret and BMC
SecretCRs for the managed cluster. The pull secret must contain all the credentials necessary for installing OpenShift Container Platform and all required Operators. See "Creating the managed bare-metal host secrets" for more information.The secrets are referenced from the
ClusterInstancecustom resource (CR) by name. The namespace must match theClusterInstancenamespace. -
Create a
ClusterInstanceCR for your cluster in your local clone of the Git repository:-
Choose the appropriate example for your CR from the
out/argocd/example/clusterinstance/folder. The folder includes example files for single node, three-node, and standard clusters:-
example-sno.yaml -
example-3node.yaml -
example-standard.yaml
-
-
Change the cluster and host details in the example file to match the type of cluster you want. For example:
Example single-node OpenShift ClusterInstance CR# example-node1-bmh-secret & assisted-deployment-pull-secret need to be created under same namespace example-ai-sno --- apiVersion: siteconfig.open-cluster-management.io/v1alpha1 kind: ClusterInstance metadata: name: "example-ai-sno" namespace: "example-ai-sno" spec: baseDomain: "example.com" pullSecretRef: name: "assisted-deployment-pull-secret" clusterImageSetNameRef: "openshift-4.22" sshPublicKey: "ssh-rsa AAAA..." clusterName: "example-ai-sno" networkType: "OVNKubernetes" # installConfigOverrides is a generic way of passing install-config # parameters through the siteConfig. The 'capabilities' field configures # the composable openshift feature. In this 'capabilities' setting, we # remove all the optional set of components. # Notes: # - OperatorLifecycleManager is needed for 4.15 and later # - NodeTuning is needed for 4.13 and later, not for 4.12 and earlier # - Ingress is needed for 4.16 and later installConfigOverrides: | { "capabilities": { "baselineCapabilitySet": "None", "additionalEnabledCapabilities": [ "NodeTuning", "OperatorLifecycleManager", "Ingress" ] } } # Include references to extraManifest ConfigMaps. extraManifestsRefs: - name: sno-extra-manifest-configmap extraLabels: ManagedCluster: # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples du-profile: "latest" # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples in ../policygentemplates: # ../policygentemplates/common-ranGen.yaml will apply to all clusters with 'common: true' common: "true" # ../policygentemplates/group-du-sno-ranGen.yaml will apply to all clusters with 'group-du-sno: ""' group-du-sno: "" # ../policygentemplates/example-sno-site.yaml will apply to all clusters with 'sites: "example-sno"' # Normally this should match or contain the cluster name so it only applies to a single cluster sites : "example-sno" clusterNetwork: - cidr: 1001:1::/48 hostPrefix: 64 machineNetwork: - cidr: 1111:2222:3333:4444::/64 serviceNetwork: - cidr: 1001:2::/112 additionalNTPSources: - 1111:2222:3333:4444::2 # Initiates the cluster for workload partitioning. Setting specific reserved/isolated CPUSets is done via PolicyTemplate # please see Workload Partitioning Feature for a complete guide. cpuPartitioningMode: AllNodes templateRefs: - name: ai-cluster-templates-v1 namespace: open-cluster-management nodes: - hostName: "example-node1.example.com" role: "master" bmcAddress: "idrac-virtualmedia+https://[1111:2222:3333:4444::bbbb:1]/redfish/v1/Systems/System.Embedded.1" bmcCredentialsName: name: "example-node1-bmh-secret" bootMACAddress: "AA:BB:CC:DD:EE:11" # Use UEFISecureBoot to enable secure boot, UEFI to disable. bootMode: "UEFISecureBoot" rootDeviceHints: deviceName: "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0" # disk partition at `/var/lib/containers` with ignitionConfigOverride. Some values must be updated. See DiskPartitionContainer.md in argocd folder for more details ignitionConfigOverride: | { "ignition": { "version": "3.2.0" }, "storage": { "disks": [ { "device": "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0", "partitions": [ { "label": "var-lib-containers", "sizeMiB": 0, "startMiB": 250000 } ], "wipeTable": false } ], "filesystems": [ { "device": "/dev/disk/by-partlabel/var-lib-containers", "format": "xfs", "mountOptions": [ "defaults", "prjquota" ], "path": "/var/lib/containers", "wipeFilesystem": true } ] }, "systemd": { "units": [ { "contents": "# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target", "enabled": true, "name": "var-lib-containers.mount" } ] } } nodeNetwork: interfaces: - name: eno1 macAddress: "AA:BB:CC:DD:EE:11" config: interfaces: - name: eno1 type: ethernet state: up ipv4: enabled: false ipv6: enabled: true address: # For SNO sites with static IP addresses, the node-specific, # API and Ingress IPs should all be the same and configured on # the interface - ip: 1111:2222:3333:4444::aaaa:1 prefix-length: 64 dns-resolver: config: search: - example.com server: - 1111:2222:3333:4444::2 routes: config: - destination: ::/0 next-hop-interface: eno1 next-hop-address: 1111:2222:3333:4444::1 table-id: 254 templateRefs: - name: ai-node-templates-v1 namespace: open-cluster-managementFor more information about BMC addressing, see the "Additional resources" section. The
installConfigOverridesandignitionConfigOverridefields are expanded in the example for ease of readability.To override the default
BareMetalHostCR for a node, create a custom node template in aConfigMapand reference it in the node-levelspec.nodes.templateRefsfield in theClusterInstanceCR. Ensure that you set theargocd.argoproj.io/sync-wave: "3"annotation in your overrideBareMetalHostCR. -
You can inspect the default set of extra-manifest
MachineConfigCRs inout/argocd/extra-manifest. It is automatically applied to the cluster when it is installed. -
Optional: To provision additional install-time manifests on the provisioned cluster, package your extra manifest CRs in a
ConfigMapand reference it in theextraManifestsRefsfield of theClusterInstanceCR. For more information, see "Customizing extra installation manifests in the GitOps ZTP pipeline".For optimal cluster performance, enable crun for master and worker nodes in single-node OpenShift, single-node OpenShift with additional worker nodes, three-node OpenShift, and standard clusters.
Enable crun in a
ContainerRuntimeConfigCR as an additional Day 0 install-time manifest to avoid the cluster having to reboot.The
enable-crun-master.yamlandenable-crun-worker.yamlCR files are in theout/source-crs/optional-extra-manifest/folder that you can extract from theztp-site-generatecontainer.
-
-
Add the
ClusterInstanceCR to thekustomization.yamlfile in thegeneratorssection, similar to the example shown inout/argocd/example/clusterinstance/kustomization.yaml. -
Commit the
ClusterInstanceCR and associatedkustomization.yamlchanges in your Git repository and push the changes.The ArgoCD pipeline detects the changes and begins the managed cluster deployment.
-
Verify that the custom roles and labels are applied after the node is deployed:
$ oc describe node example-node.example.comThe following example output shows the custom roles and labels:
Name: example-node.example.com Roles: control-plane,example-label,master,worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux custom-label/parameter1=true kubernetes.io/arch=amd64 kubernetes.io/hostname=cnfdf03.telco5gran.eng.rdu2.redhat.com kubernetes.io/os=linux node-role.kubernetes.io/control-plane= node-role.kubernetes.io/example-label= node-role.kubernetes.io/master= node-role.kubernetes.io/worker= node.openshift.io/os_id=rhcos-
node-role.kubernetes.io/example-label=shows the custom label applied to the node.
-