Add a new RHCOS worker node with a custom /var partition in AWS

You can add an RHCOS worker node with a custom /var partition in AWS by creating a user data secret and a compute machine set that use the required device naming schema.

OpenShift Container Platform supports partitioning devices during installation by using machine configs that are processed during the bootstrap. However, if you use /var partitioning, the device name must be determined at installation and cannot be changed. You cannot add different instance types as nodes if they have a different device naming schema. For example, if you configured the /var partition with the default AWS device name for m4.large instances, dev/xvdb, you cannot directly add an AWS m5.large instance, as m5.large instances use a /dev/nvme1n1 device by default. The device might fail to partition due to the different naming schema.

The procedure in this section shows how to add a new Red Hat Enterprise Linux CoreOS (RHCOS) compute node with an instance that uses a different device name from what was configured at installation. You create a custom user data secret and configure a new compute machine set. These steps are specific to an AWS cluster. The principles apply to other cloud deployments also. However, the device naming schema is different for other deployments and should be determined on a per-case basis.

Procedure

  1. On a command line, change to the openshift-machine-api namespace:

    $ oc project openshift-machine-api

  2. Create a new secret from the worker-user-data secret:

    1. Export the userData section of the secret to a text file:

      $ oc get secret worker-user-data --template='{{index .data.userData | base64decode}}' | jq > userData.txt

    2. Edit the text file to add the storage, filesystems, and systemd stanzas for the partitions you want to use for the new node. You can specify any Ignition configuration parameters as needed.

      Note

      Do not change the values in the ignition stanza.

      		 
      {
  "ignition": {
    "config": {
      "merge": [
        {
          "source": "https:...."
        }
      ]
    },
    "security": {
      "tls": {
        "certificateAuthorities": [
          {
            "source": "data:text/plain;charset=utf-8;base64,.....=="
          }
        ]
      }
    },
    "version": "3.2.0"
  },
  "storage": {
    "disks": [
      {
        "device": "/dev/nvme1n1",
        "partitions": [
          {
            "label": "var",
            "sizeMiB": 50000,
            "startMiB": 0
          }
        ]
      }
    ],
    "filesystems": [
      {
        "device": "/dev/disk/by-partlabel/var",
        "format": "xfs",
        "path": "/var"
      }
    ]
  },
  "systemd": {
    "units": [
      {
        "contents": "[Unit]\nBefore=local-fs.target\n[Mount]\nWhere=/var\nWhat=/dev/disk/by-partlabel/var\nOptions=defaults,pquota\n[Install]\nWantedBy=local-fs.target\n",
        "enabled": true,
        "name": "var.mount"
      }
    ]
  }
}

      where:

      device (under disks)

      Specifies an absolute path to the AWS block device.

      sizeMiB

      Specifies the size of the data partition in Mebibytes.

      startMiB

      Specifies the start of the partition in Mebibytes. When adding a data partition to the boot disk, a minimum value of 25000 MB (Mebibytes) is recommended. The root file system is automatically resized to fill all available space up to the specified offset. If no value is specified, or if the specified value is smaller than the recommended minimum, the resulting root file system will be too small, and future reinstalls of RHCOS might overwrite the beginning of the data partition.

      device (under filesystems)

      Specifies an absolute path to the /var partition.

      format

      Specifies the filesystem format.

      path

      Specifies the mount-point of the filesystem while Ignition is running relative to where the root filesystem will be mounted. This is not necessarily the same as where it should be mounted in the real root, but it is encouraged to make it the same.

      units

      Defines a systemd mount unit that mounts the /dev/disk/by-partlabel/var device to the /var partition.

    3. Extract the disableTemplating section from the work-user-data secret to a text file:

      $ oc get secret worker-user-data --template='{{index .data.disableTemplating | base64decode}}' | jq > disableTemplating.txt

    4. Create the new user data secret file from the two text files. This user data secret passes the additional node partition information in the userData.txt file to the newly created node.

      $ oc create secret generic worker-user-data-x5 --from-file=userData=userData.txt --from-file=disableTemplating=disableTemplating.txt

  3. Create a new compute machine set for the new node:

    1. Create a new compute machine set YAML file, similar to the following, which is configured for AWS. Add the required partitions and the newly-created user data secret:

      Tip

      Use an existing compute machine set as a template and change the parameters as needed for the new node.

      		 
      apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: auto-52-92tf4
  name: worker-us-east-2-nvme1n1
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: auto-52-92tf4
      machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: auto-52-92tf4
        machine.openshift.io/cluster-api-machine-role: worker
        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
    spec:
      metadata: {}
      providerSpec:
        value:
          ami:
            id: ami-0c2dbd95931a
          apiVersion: awsproviderconfig.openshift.io/v1beta1
          blockDevices:
          - DeviceName: /dev/nvme1n1
            ebs:
              encrypted: true
              iops: 0
              volumeSize: 120
              volumeType: gp2
          - DeviceName: /dev/nvme1n2
            ebs:
              encrypted: true
              iops: 0
              volumeSize: 50
              volumeType: gp2
          credentialsSecret:
            name: aws-cloud-credentials
          deviceIndex: 0
          iamInstanceProfile:
            id: auto-52-92tf4-worker-profile
          instanceType: m6i.large
          kind: AWSMachineProviderConfig
          metadata:
            creationTimestamp: null
          placement:
            availabilityZone: us-east-2b
            region: us-east-2
          securityGroups:
          - filters:
            - name: tag:Name
              values:
              - auto-52-92tf4-worker-sg
          subnet:
            id: subnet-07a90e5db1
          tags:
          - name: kubernetes.io/cluster/auto-52-92tf4
            value: owned
          userDataSecret:
            name: worker-user-data-x5

      where:

      name

      Specifies a name for the new node.

      DeviceName: /dev/nvme1n1

      Specifies an absolute path to the AWS block device, here an encrypted EBS volume.

      DeviceName: /dev/nvme1n2

      Optional. Specifies an additional EBS volume.

      userDataSecret.name

      Specifies the user data secret file.

    2. Create the compute machine set:

      $ oc create -f <file-name>.yaml

      The machines might take a few moments to become available.

  4. Verify that the new partition and nodes are created:

    1. Verify that the compute machine set is created:

      $ oc get machineset
      Example output
      NAME                                               DESIRED   CURRENT   READY   AVAILABLE   AGE
ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1a        1         1         1       1           124m
ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1b        2         2         2       2           124m
worker-us-east-2-nvme1n1                           1         1         1       1           2m35s

      The worker-us-east-2-nvme1n1 compute machine set is the new compute machine set.

    2. Verify that the new node is created:

      $ oc get nodes
      Example output
      NAME                           STATUS   ROLES    AGE     VERSION
ip-10-0-128-78.ec2.internal    Ready    worker   117m    v1.35.4
ip-10-0-146-113.ec2.internal   Ready    master   127m    v1.35.4
ip-10-0-153-35.ec2.internal    Ready    worker   118m    v1.35.4
ip-10-0-176-58.ec2.internal    Ready    master   126m    v1.35.4
ip-10-0-217-135.ec2.internal   Ready    worker   2m57s   v1.35.4
ip-10-0-225-248.ec2.internal   Ready    master   127m    v1.35.4
ip-10-0-245-59.ec2.internal    Ready    worker   116m    v1.35.4

      The ip-10-0-217-135.ec2.internal node is the new node.

    3. Verify that the custom /var partition is created on the new node:

      $ oc debug node/<node-name> -- chroot /host lsblk

      For example:

      $ oc debug node/ip-10-0-217-135.ec2.internal -- chroot /host lsblk
      Example output
      NAME        MAJ:MIN  RM  SIZE RO TYPE MOUNTPOINT
nvme0n1     202:0    0   120G  0 disk
|-nvme0n1p1 202:1    0     1M  0 part
|-nvme0n1p2 202:2    0   127M  0 part
|-nvme0n1p3 202:3    0   384M  0 part /boot
`-nvme0n1p4 202:4    0 119.5G  0 part /sysroot
nvme1n1     202:16   0    50G  0 disk
`-nvme1n1p1 202:17   0  48.8G  0 part /var

      The nvme1n1 device is mounted to the /var partition.

← Previous: Override storage or partition setup | ↑ Up: Change the installed disk-partition schema for new nodes. | ⌂ Home: OpenShift Container Platform | Next: Disk partitioning →