Red Hat build of MicroShift

Storage

Understand LVMS storage

MicroShift uses the Logical Volume Manager Storage (LVMS) plugin to provide dynamic, persistent storage for workloads. LVMS provisions local storage on the device and manages storage classes, device classes, and thin volumes. Understanding how LVMS works and what resources it manages helps you plan and configure storage for your MicroShift deployments.

Dynamic storage with the LVMS plugin

MicroShift provides dynamic storage provisioning that is ready for immediate use with the logical volume manager storage (LVMS) Container Storage Interface (CSI) provider. Dynamic provisioning creates storage volumes on-demand, eliminating the need for pre-provisioned storage.

Ephemeral storage

Pods and containers are ephemeral or transient in nature and designed for stateless applications. Ephemeral storage allows administrators and developers to better manage the local storage for some of their operations.

Persistent storage

Persistent storage in MicroShift enables stateful applications to retain data beyond the lifecycle of individual pods. You can use persistent volumes (PVs) to provision storage and persistent volume claims (PVCs) to request storage for your applications.

Dynamic storage provisioning

In MicroShift, dynamic storage provisioning creates persistent volumes when applications request storage. You can scale your storage workloads on-demand, without manually preallocating storage.

LVMS system requirements

To prepare your infrastructure for storage operations, review the system specifications for using LVMS in MicroShift. Verifying these requirements ensures your environment meets the necessary resource standards for a successful deployment.

Volume group name

If you did not configure LVMS in an lvmd.yaml file placed in the /etc/microshift/ directory, MicroShift attempts to assign a default volume group (VG) dynamically by running the vgs command.

MicroShift assigns a default VG when only one VG is found.
If more than one VG is present, the VG named microshift is assigned as the default.
If a VG named microshift does not exist, LVMS is not deployed.

If there are no volume groups on the MicroShift host, LVMS is disabled.

If you want to use a specific VG, LVMS must be configured to select that VG. You can change the default name of the VG in the configuration file. For details, read the "Configuring the LVMS" section of this document.

You can change the default name of the VG in the configuration file. For details, read the "Configuring the LVMS" section of this document.

After MicroShift starts, you can update the lvmd.yaml to include or remove VGs. To implement changes, you must restart MicroShift. If the lvmd.yaml is deleted, MicroShift attempts to find a default VG again.

Volume size increments

The LVMS provisions storage in increments of 1 gigabyte (GB). Storage requests are rounded up to the nearest GB. When the capacity of a VG is less than 1 GB, the PersistentVolumeClaim registers a ProvisioningFailed event, for example:

Example output

Warning  ProvisioningFailed    3s (x2 over 5s)  topolvm.cybozu.com_topolvm-controller-858c78d96c-xttzp_0fa83aef-2070-4ae2-bcb9-163f818dcd9f failed to provision volume with
StorageClass "topolvm-provisioner": rpc error: code = ResourceExhausted desc = no enough space left on VG: free=(BYTES_INT), requested=(BYTES_INT)

LVMS deployment

To ensure local storage is ready for use, MicroShift automatically deploys LVMS into the openshift-storage namespace at startup. This automated process prepares the node for storage operations immediately, eliminating the need for manual installation.

LVMS uses StorageCapacity tracking to ensure that pods with an LVMS PVC are not scheduled if the requested storage is greater than the free storage of the volume group. For more information about StorageCapacity tracking, see "Storage Capacity".

Limitations to configure the size of the devices used in LVM Storage

To ensure your devices are compatible with storage operations, review the size configuration limitations in LVM Storage. Adhering to these constraints prevents provisioning failures by ensuring selected devices meet the required capacity specifications.

When provisioning storage by using LVM Storage, the following factors limit device size:

The total storage size that you can provision is limited by the size of the underlying Logical Volume Manager (LVM) thin pool and the over-provisioning factor.
The size of the logical volume depends on the size of the Physical Extent (PE) and the Logical Extent (LE).
- You can define the size of PE and LE during the physical and logical device creation.
- The default PE and LE size is 4 MiB.
- If the size of the PE is increased, the maximum size of the LVM is determined by the kernel limits and your disk space.

The following tables describe the chunk size and volume size limits for static and host configurations:

Table 16. Tested configuration
Parameter	Value
Chunk size	128 KiB
Maximum volume size	32 TiB

Table 17. Theoretical size limits for static configuration
Parameter	Minimum value	Maximum value
Chunk size	64 KiB	1 GiB
Volume size	Minimum size of the underlying Red Hat Enterprise Linux CoreOS (RHCOS) system.	Maximum size of the underlying RHCOS system.

Table 18. Theoretical size limits for a host configuration
Parameter	Value
Chunk size	This value is based on the configuration in the `lvm.conf` file. By default, the configuration sets the value to `128` KiB.
Maximum volume size	Equal to the maximum volume size of the underlying RHCOS system.
Minimum volume size	Equal to the minimum volume size of the underlying RHCOS system.

Creating an LVMS configuration file

To customize storage settings, create an LVMS configuration file named lvmd.yaml. You must place this file in the /etc/microshift/ directory to ensure MicroShift detects and applies your configuration at startup.

Procedure

To create the lvmd.yaml configuration file, run the following command:

$ sudo cp /etc/microshift/lvmd.yaml.default /etc/microshift/lvmd.yaml

Basic LVMS configuration example

To customize storage operations, pass through your LVM configuration to MicroShift. With this flexibility, you can define custom volume groups, thin volume provisioning parameters, and reserved unallocated space by editing the LVMS configuration file.

You must restart MicroShift to deploy configuration changes after editing the file.

If you need to take volume snapshots, you must use thin provisioning in your lvmd.conf file. If you do not need to take volume snapshots, you can use thick volumes.

The following lvmd.yaml example file shows a basic LVMS configuration:

LVMS configuration example

socket-name:
device-classes:
  - name: "default"
    volume-group: "VGNAMEHERE"
    spare-gb: 0 
    default:

+ where:

socket-name: Specifies the UNIX domain socket endpoint of gRPC. Defaults to /run/lvmd/lvmd.socket. Takes a string value.
device-classes: Specifies a list of maps for the settings for each device-class.
device-classes.name: Specifies the name of the device-class. Takes a string value.
device-classes.volume-group: Specifies the group where the device-class creates the logical volumes. Takes a string value.
device-classes.spare-gb: Specifies the storage capacity in GB to be left unallocated in the volume group. Defaults to 0. Takes an unsigned 64-bit integer.
device-classes.default: Specifies that the device-class is used by default. Defaults to false. At least one value must be entered in the YAML file when this value is set to true. Takes a boolean value.

A race condition prevents LVMS from accurately tracking the allocated space and preserving the spare-gb for a device class when multiple PVCs are created simultaneously. Use separate volume groups and device classes to protect the storage of highly dynamic workloads from each other.

Using the LVMS

To automatically provision and mount a logical volume to a pod, use the LVMS default StorageClass. By creating a PersistentVolumeClaim object without defining the .spec.storageClassName field, you trigger the dynamic provisioning of a PersistentVolume from this default resource.

Use the following procedure to provision and mount a logical volume to a pod.

Procedure

To provision and mount a logical volume to a pod, run the following command:

$ cat <<EOF | oc apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: my-lv-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1G
---
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: nginx
    image: nginx
    command: ["/usr/bin/sh", "-c"]
    args: ["sleep", "1h"]
    volumeMounts:
    - mountPath: /mnt
      name: my-volume
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault
  volumes:
    - name: my-volume
      persistentVolumeClaim:
        claimName: my-lv-pvc
EOF

Device classes

To define custom storage groups, create custom device classes by adding a device-classes array to your logical volume manager storage (LVMS) configuration. With this configuration, you can enable MicroShift to categorize devices based on your specific storage requirements.

Add the array to the /etc/microshift/lvmd.yaml configuration file. A single device class must be set as the default. You must restart MicroShift for configuration changes to take effect.

Removing a device class while there are still persistent volumes or VolumeSnapshotContent objects connected to that device class breaks both thick and thin provisioning.

You can define multiple device classes in the device-classes array. These classes can be a mix of thick and thin volume configurations.

Example of a mixed device-class array

socket-name: /run/topolvm/lvmd.sock
device-classes:
  - name: ssd
    volume-group: ssd-vg
    spare-gb: 0
    default: true
  - name: hdd
    volume-group: hdd-vg
    spare-gb: 0
  - name: thin
    spare-gb: 0
    thin-pool:
      name: thin
      overprovision-ratio: 10
    type: thin
    volume-group: ssd
  - name: striped
    volume-group: multi-pv-vg
    spare-gb: 0
    stripe: 2
    stripe-size: "64"
    lvcreate-options:

device-classes.spare-gb`: Specifies the spare capacity. When you set this value to anything other than 0, more space can be allocated than expected.
device-classes.lvcreate-options: Specifies extra arguments to pass to the lvcreate command, such as --type=<type>. Neither MicroShift nor the LVMS verifies lvcreate-options values. These optional values are passed as is to the lvcreate command. Ensure that the options specified here are correct.

Storage classes

To configure the workload layer interface for device class selection, review the supported storage class parameters in MicroShift. By understanding these parameters, you can define how storage is provisioned and managed for your specific workload requirements.

The following storage class parameters are supported in MicroShift:

The csi.storage.k8s.io/fstype parameter selects the file system types. Both xfs and ext4 file system types are supported.
The topolvm.io/device-class parameter is the name of the device class. If a device class is not provided, the default device class is assumed.

Multiple storage classes can refer to the same device class. You can provide varying sets of parameters for the same backing device class, such as xfs and ext4 variants.

Example MicroShift default storage class resource

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: topolvm-provisioner
parameters:
  "csi.storage.k8s.io/fstype": "xfs"
provisioner: topolvm.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion:
# ...

where:

storageclass.kubernetes.io/is-default-class: Specifies an example of the default storage class. If a PVC does not specify a storage class, this class is assumed. There can only be one default storage class in a MicroShift node. Having no value assigned to this annotation is also supported.
csi.storage.k8s.io/fstype: Specifies what file system to provision on the volume. Options are "xfs" and "ext4".
provisioner: Specifies what provisioner should manage this class.
volumeBindingMode: Specifies whether to provision the volume before a client pod is present or immediately. Options are WaitForFirstConsumer and Immediate. WaitForFirstConsumer is recommended to ensure that storage is only provisioned for pods that can be scheduled.
allowVolumeExpansion: Specifies if PVCs provisioned from the StorageClass permit expansion. The MicroShift LVMS CSI plugin does support volume expansion, but if this value is set to false, expansion is blocked.

Persistent storage overview

Stateful applications deployed in containers require persistent storage. MicroShift uses a pre-provisioned storage framework called persistent volumes (PV) to allow node administrators to provision persistent storage. The data inside these volumes can exist beyond the lifecycle of an individual pod. Developers can use persistent volume claims (PVCs) to request storage requirements.

Managing storage is a distinct problem from managing compute resources. {product-title} uses the Kubernetes persistent volume (PV) framework to allow cluster administrators to provision persistent storage for a cluster. Developers can use persistent volume claims (PVCs) to request PV resources without having specific knowledge of the underlying storage infrastructure.

PVCs are specific to a project, and are created and used by developers as a means to use a PV. PV resources on their own are not scoped to any single project; they can be shared across the entire {product-title} node and claimed from any project. After a PV is bound to a PVC, that PV can not then be bound to additional PVCs. This has the effect of scoping a bound PV to a single namespace, that of the binding project.

PVs are defined by a PersistentVolume API object, which represents a piece of existing storage in the cluster that was either statically provisioned by the cluster administrator or dynamically provisioned using a StorageClass object. It is a resource in the cluster just like a node is a cluster resource.

PVs are volume plugins like Volumes but have a lifecycle that is independent of any individual pod that uses the PV. PV objects capture the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.

High availability of storage in the infrastructure is left to the underlying storage provider.

PVCs are defined by a PersistentVolumeClaim API object, which represents a request for storage by a developer. It is similar to a pod in that pods consume node resources and PVCs consume PV resources. For example, pods can request specific levels of resources, such as CPU and memory, while PVCs can request specific storage capacity and access modes. For example, they can be mounted once read-write or many times read-only.

Use persistent storage

Persistent storage on MicroShift lets applications retain data across pod restarts and node reboots. You can create and manage persistent volume claims (PVCs), control access modes, and use LVMS-backed thin volumes to provide durable storage to your workloads.

Lifecycle of a volume and claim

The persistent volume (PV) lifecycle follows five phases: provision, bind, use, release, and reclaim. Each phase has behaviors that affect storage availability and data retention. Understanding the lifecycle helps you choose reclaim policies, troubleshoot binding failures, and prevent data loss.

PVs are resources in the cluster. Persistent volume claims (PVCs) are requests for those resources and also act as claim checks to the resource.

The interaction between PVs and PVCs has the following lifecycle.

Provision storage

In response to requests from a developer defined in a PVC, a cluster administrator configures one or more dynamic provisioners that provision storage and a matching PV.

Bind claims

When you create a PVC, you request a specific amount of storage, specify the required access mode, and create a storage class to describe and classify the storage. The control loop in the master watches for new PVCs and binds the new PVC to an appropriate PV. If an appropriate PV does not exist, a provisioner for the storage class creates one.

The size of all PVs might exceed your PVC size. This is especially true with manually provisioned PVs. To minimize the excess,{product-title} binds to the smallest PV that matches all other criteria.

Claims remain unbound indefinitely if a matching volume does not exist or cannot be created with any available provisioner servicing a storage class. Claims are bound as matching volumes become available. For example, a cluster with many manually provisioned 50Gi volumes would not match a PVC requesting 100Gi. The PVC can be bound when a 100Gi PV is added to the cluster.

Use pods and claimed PVs

Pods use claims as volumes. The cluster inspects the claim to find the bound volume and mounts that volume for a pod. For those volumes that support multiple access modes, you must specify which mode applies when you use the claim as a volume in a pod.

After you have a claim, and that claim is bound, the bound PV belongs to you for as long as you need it. You can schedule pods and access claimed PVs by including persistentVolumeClaim in the pod’s volumes block.

If you attach persistent volumes that have high file counts to pods, those pods can fail or can take a long time to start. For more information, see the Red Hat Knowledgebase article "When using Persistent Volumes with high file counts in OpenShift, why do pods fail to start or take an excessive amount of time to achieve "Ready" state?".

Release a persistent volume

When you are finished with a volume, you can delete the PVC object from the API, which allows reclamation of the resource. The volume is considered released when the claim is deleted, but it is not yet available for another claim. The previous claimant’s data remains on the volume and must be handled according to policy.

Reclaim policy for persistent volumes

The reclaim policy of a persistent volume tells the cluster what to do with the volume after it is released. A volume’s reclaim policy can be Retain, Recycle, or Delete.

Retain reclaim policy allows manual reclamation of the resource for those volume plugins that support it.
Recycle reclaim policy recycles the volume back into the pool of unbound persistent volumes once it is released from its claim.

The Recycle reclaim policy is deprecated in {product-title} 4. Dynamic provisioning is recommended for equivalent and better functionality.

Delete reclaim policy deletes both the PersistentVolume object from {product-title} and the associated storage asset in external infrastructure, such as Amazon Elastic Block Store (Amazon EBS) or VMware vSphere.

Dynamically provisioned volumes are always deleted.

Reclaiming a persistent volume manually

Manually reclaim released persistent volumes (PVs) to make them available for new claims or to properly clean up storage assets.

When a persistent volume claim (PVC) is deleted, the persistent volume (PV) still exists and is considered "released". However, the PV is not yet available for another claim because the data of the previous claimant remains on the volume.

Procedure

Delete the persistent volume (PV) by running the following command:
```
$ oc delete pv <pv_name>
```
The associated storage asset in the external infrastructure, such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume, still exists after the PV is deleted.
Clean up the data on the associated storage asset.
Delete the associated storage asset. Alternately, to reuse the same storage asset, create a new PV with the storage asset definition.

Result

The reclaimed PV is now available for use by another PVC.

Changing the reclaim policy of a persistent volume

Change a persistent volume’s reclaim policy to control whether storage is automatically deleted or retained when claims are removed. Switching from Delete to Retain protects data from accidental loss, while changing to Delete enables automatic cleanup of unused volumes.

Procedure

List the persistent volumes in your cluster:

$ oc get pv

Example output

NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM             STORAGECLASS     REASON    AGE
 pvc-b6efd8da-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim1    manual                     10s
 pvc-b95650f8-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim2    manual                     6s
 pvc-bb3ca71d-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim3    manual                     3s

Choose one of your persistent volumes and change its reclaim policy:

$ oc patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Verify that your chosen persistent volume has the right policy:

$ oc get pv

Example output

NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM             STORAGECLASS     REASON    AGE
 pvc-b6efd8da-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim1    manual                     10s
 pvc-b95650f8-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim2    manual                     6s
 pvc-bb3ca71d-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Retain          Bound     default/claim3    manual                     3s

In the preceding output, the volume bound to claim default/claim3 now has a Retain reclaim policy. The volume will not be automatically deleted when a user deletes claim default/claim3.

Persistent volumes

Configure persistent volumes (PVs) with capacity, access modes, mount options, and reclaim policies to manage cluster-wide storage resources across their lifecycle phases.

Each storage backend supports different access mode combinations, and volumes transition through phases (Available, Bound, Released, Failed) affecting claim availability.

Each PV contains a spec and status, which is the specification and status of the volume, for example:

Example PersistentVolume object definition

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  ...
status:
  ...

metadata.name: Specifies the name of the persistent volume.
spec.storage: Specifies the amount of storage available to the volume.
spec.accessModes: Specifies the access mode, defining the read/write and mount permissions.
spec.persistentVolumeReclaimPolicy: Specifies the reclaim policy, indicating how the resource should be handled once it is released.

You can view the name of a PVC that is bound to a PV by running the following command:

$ oc get pv <pv_name> -o jsonpath='{.spec.claimRef.name}'

Types of PVs

+ {product-title} supports the following persistent volume plugins:

AWS Elastic Block Store (EBS), which is installed by default.
GCP Persistent Disk
GCP Filestore

Capacity

Generally, a persistent volume (PV) has a specific storage capacity. This is set by using the capacity attribute of the PV.

Currently, storage capacity is the only resource that can be set or requested. Future attributes may include IOPS, throughput, and so on.

Access modes

A persistent volume can be mounted on a host in any way supported by the resource provider. Providers have different capabilities and each PV’s access modes are set to the specific modes supported by that particular volume. For example, NFS can support multiple read/write clients, but a specific NFS PV might be exported on the server as read-only. Each PV gets its own set of access modes describing that specific PV’s capabilities.

Claims are matched to volumes with similar access modes. The only two matching criteria are access modes and size. A claim’s access modes represent a request. Therefore, you might be granted more, but never less. For example, if a claim requests RWO, but the only volume available is an NFS PV (RWO+ROX+RWX), the claim would then match NFS because it supports RWO.

Direct matches are always attempted first. The volume’s modes must match or contain more modes than you requested. The size must be greater than or equal to what is expected. If two types of volumes, such as NFS and iSCSI, have the same set of access modes, either of them can match a claim with those modes. There is no ordering between types of volumes and no way to choose one type over another.

All volumes with the same modes are grouped, and then sorted by size, smallest to largest. The binder gets the group with matching modes and iterates over each, in size order, until one size matches.

Volume access modes describe volume capabilities. They are not enforced constraints. The storage provider is responsible for runtime errors resulting from invalid use of the resource. Errors in the provider show up at runtime as mount errors.

For example, NFS offers ReadWriteOnce access mode. If you want to use the volume’s ROX capability, mark the claims as ReadOnlyMany.

iSCSI and Fibre Channel volumes do not currently have any fencing mechanisms. You must ensure the volumes are only used by one node at a time. In certain situations, such as draining a node, the volumes can be used simultaneously by two nodes. Before draining the node, delete the pods that use the volumes.

The following table lists the access modes:

Table 19. Access modes
Access Mode	CLI abbreviation	Description
ReadWriteOnce	`RWO`	The volume can be mounted as read/write by a single node.
ReadWriteOncePod	`RWOP`	The volume can be mounted as read/write by a single pod on a single node.

Table 20. Supported access modes for persistent volumes
Volume plugin	ReadWriteOnce ^[1]	ReadWriteOncePod	ReadOnlyMany	ReadWriteMany
AWS EBS ^[2]	✅	✅
AWS EFS	✅	✅	✅	✅
GCP Persistent Disk	✅ ^[4]	✅	✅	✅ ^[4]
GCP Filestore	✅	✅	✅	✅
LVM Storage	✅	✅

ReadWriteOnce (RWO) volumes cannot be mounted on multiple nodes. If a node fails, the system does not allow the attached RWO volume to be mounted on a new node because it is already assigned to the failed node. If you encounter a multi-attach error message as a result, force delete the pod on a shutdown or crashed node to avoid data loss in critical workloads, such as when dynamic persistent volumes are attached.
Use a recreate deployment strategy for pods that rely on AWS EBS.
Only raw block volumes support the ReadWriteMany (RWX) access mode for Fibre Channel and iSCSI. For more information, see "Block volume support".
For GCP hyperdisk-balanced disks:
- The supported access modes are:
  - ReadWriteOnce
  - ReadWriteMany
- Cloning and snapshotting is disabled for disks with ReadWriteMany access mode enabled.
- You can attach a single hyperdisk-balanced disk volume in ReadWriteMany to a maximum of 8 instances.
- You can only resize a disk in ReadWriteMany if you detach the disk from all instances.
- For additional limitations, see Google Cloud documentation "GCP hyperdisk-balanced disk additional limitations".
If the underlying vSphere environment supports the vSAN file service, the vSphere Container Storage Interface (CSI) Driver Operator installed by {product-title} supports provisioning of ReadWriteMany (RWX) volumes. If you do not have vSAN file service configured, and you request RWX, the volume fails to get created and an error is logged. For more information, see "VMware vSphere CSI Driver Operator".

Phase

Volumes can be found in one of the following phases:

+ .Volume phases

Phase	Description
Available	A free resource not yet bound to a claim.
Bound	The volume is bound to a claim.
Released	The claim was deleted, but the resource is not yet reclaimed by the cluster.
Failed	The volume has failed its automatic reclamation.

Phase

Description

Available

A free resource not yet bound to a claim.

Bound

The volume is bound to a claim.

Released

The claim was deleted, but the resource is not yet reclaimed by the cluster.

Failed

The volume has failed its automatic reclamation.

Last phase transition time

The LastPhaseTransitionTime field has a timestamp that updates every time a persistent volume (PV) transitions to a different phase (pv.Status.Phase). To find the time of the last phase transition for a PV, run the following command:

$ oc get pv <pv_name> -o json | jq '.status.lastPhaseTransitionTime'

For '.status.lastPhaseTransitionTime' specify the name of the PV that you want to see the last phase transition.

Mount options

You can specify mount options while mounting a PV by using the attribute mountOptions.

For example:

Mount options example

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  mountOptions:
    - nfsvers=4.1
  nfs:
    path: /tmp
    server: 172.17.0.2
  persistentVolumeReclaimPolicy: Retain
  claimRef:
    name: claim1
    namespace: default

spec.mountOptions: Specified mount options are used while mounting the PV to the disk.

The following PV types support mount options:

AWS Elastic Block Store (EBS)
AWS Elastic File Storage (EFS)
Azure Disk
Azure File
Cinder
GCE Persistent Disk
iSCSI
Local volume
NFS
Red Hat OpenShift Data Foundation (Ceph RBD only)
CIFS/SMB
VMware vSphere

Fibre Channel and HostPath PVs do not support mount options.

Persistent volumes with RWO access mode permissions

To enable concurrent access for pods on a single node, configure the ReadWriteOnce (RWO) access mode for your Persistent Volume Claims (PVCs). This setting allows multiple workloads on the same node to read from and write to the same Persistent Volume (PV) simultaneously.

Sometimes pods of the same node are not able to read or write into the same PV. This happens when the pods in the node do not have the same SELinux context.

Persistent volumes can be mounted, while later claimed by PVCs, with the RWO access mode.

Control permissions with security context constraints

You can use security context constraints (SCCs) to control permissions for the pods in your node. These permissions determine the actions that a pod can perform and what resources it can access. You can use SCCs to define a set of conditions that a pod must run with to be accepted into the system.

For more information, see "Managing security context constraints".

Only RWO volume mounts are supported. SCC could be blocked if pods are not operating with the SCC contexts.

About LVM thin volumes

To enable advanced storage capabilities, such as volume snapshots and volume cloning, complete specific configuration steps. Preparing your environment ensures that the necessary components are active and ready to support these features for your workloads.

The following list describes the configuration steps:

Configure both the logical volume manager storage (LVMS) provider and the node.
Provision a logical volume manager (LVM) thin-pool on the RHEL for Edge host.
Attach LVM thin-pools to a volume group.

To create Container Storage Interface (CSI) snapshots, you must configure thin volumes on the RHEL for Edge host. The CSI does not support volume shrinking.

When using thin provisioning, you must monitor the storage pool and add more capacity as the available physical space runs out. You can configure the storage pool to auto expand when there is available space within the volume group (VG). See "Creating a thin logical volume".

For LVMS to manage thin logical volumes (LVs), a thin-pool device-class array must be specified in the etc/lvmd.yaml configuration file. Multiple thin-pool device classes are permitted.

If additional storage pools are configured with device classes, then additional storage classes must also exist to expose the storage pools to users and workloads. To enable dynamic provisioning on a thin-pool, a StorageClass resource must be present on the node. The StorageClass resource specifies the source device-class array in the topolvm.io/device-class parameter.

Example lvmd.yaml file that specifies a single device class for a thin-pool

socket-name:
device-classes:
  - name: thin
    default: true
    spare-gb: 0
    thin-pool:
      name: thin
      overprovision-ratio: 1
    type: thin
    volume-group: ssd

where:

socket-name: Specifies the UNIX domain socket endpoint of gRPC. Defaults to /run/lvmd/lvmd.socket. Takes a string value.
device-classes: Specifies a list of maps for the settings for each device-class.
device-classes.name: Specifies the unique name of the device-class. Takes a string value.
device-classes.spare-gb: Specifies storage capacity in GB to be left unallocated in the volume group. Defaults to 0. Takes an unsigned 64-bit integer.
thin-pool.overprovision-ratio: Specifies a float factor by which you can provision additional storage based on the available storage in the thin pool. For example, if this field is set to 10, you can provision up to 10 times the amount of available storage in the thin pool. To disable over-provisioning, set this field to 1.
type: Specifies thin provisioning is required to create volume snapshots.
volume-group: Specifies the group where the device-class creates the logical volumes. Takes a string value.

When multiple PVCs are created simultaneously, a race condition prevents LVMS from accurately tracking the allocated space and preserving the storage capacity for a device class. Use separate volume groups and device classes to protect the storage of highly dynamic workloads from each other.

Persistent volume claims

Persistent volume claims (PVCs) are namespace-scoped storage requests that specify capacity, access modes, and storage class requirements. Each claim contains a spec field defining the storage request parameters and a status field tracking the binding state and current conditions of the claim.

Example PersistentVolumeClaim object definition

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
  storageClassName: gold
status:
# ...

apiVersion: Specifies the name of the PVC.
spec.accessModes: Specifies the access mode, defining the read/write and mount permissions.
requests.storage: Specifies the amount of storage available to the PVC.
storageClassName: Specifies the name of the StorageClass required by the claim.

Storage classes

To request specific storage capabilities, define the StorageClass name in the storageClassName attribute of your PersistentVolumeClaim (PVC). This setting ensures the claim binds only to matching PersistentVolumes (PVs) or triggers dynamic provisioning if the cluster administrator has configured on-demand creation.

The Cluster Storage Operator might install a default storage class depending on the platform in use. This storage class is owned and controlled by the Operator. The storage class cannot be deleted or modified beyond defining annotations and labels. If different behavior is desired, you must define a custom storage class.

The cluster administrator can also set a default storage class for all PVCs. When you configure a default storage class, the PVC must explicitly ask for StorageClass or storageClassName annotations set to "" to be bound to a PV without a storage class.

If more than one storage class is marked as default, a PVC can only be created if the storageClassName is explicitly specified. Therefore, only set one storage class as the default.

Claims as volumes

To enable pods to access storage resources, configure Persistent Volume Claims (PVCs) as volumes. By mounting the claim to the host and into the pod, the cluster locates the backing PersistentVolume (PV) in the same namespace, ensuring the workload can read and write data effectively.

Claims use the same conventions as volumes when requesting storage with specific access modes.

Claims, such as pods, can request specific quantities of a resource. In this case, the request is for storage. The same resource model applies to volumes and claims.

Mount volume to the host and into the pod example

kind: Pod
apiVersion: v1
metadata:
  name: mypod
spec:
  containers:
    - name: myfrontend
      image: dockerfile/nginx
      volumeMounts:
      - mountPath: "/var/www/html"
        name: mypd
  volumes:
    - name: mypd
      persistentVolumeClaim:
        claimName: myclaim
# ...

where:

volumeMounts.mountPath: Specifies the path to mount the volume inside the pod.
volumeMounts.name: Specifies the name of the volume to mount. Do not mount to the container root, /, or any path that is the same in the host and the container. This can corrupt your host system if the container is sufficiently privileged, such as the host /dev/pts files. Using /host is a safe option for mounting the host.
persistentVolumeClaim.claimName: Specifies the name of the PVC, that exists in the same namespace, to use.

Setting PVC viewing permissions

To monitor storage resources, verify that you have the necessary privileges to view Persistent Volume Claim (PVC) usage statistics. Ensuring you have the correct permissions means that you can access usage data and track resource consumption effectively.

To view PVC usage statistics, you must have the necessary privileges.

Procedure

If you have admin privileges, log on to MicroShift as an admin.
If you do not have admin privileges, complete the following steps:
- Create cluster roles for the user by running the following command:
  $ oc create clusterrole routes-view --verb=get,list --resource=routes
- Add the routes-view cluster role for the user by running the following command:
  $ oc admin policy add-cluster-role-to-user routes-view _<user_name>_
- Replace <user_name> with the user name.
- Add the cluster-monitoring-view cluster role for the user by running the following command:
  $ oc admin policy add-cluster-role-to-user cluster-monitoring-view _<user_name>_
- Replace <user_name> with the user name.

Viewing PVC usage statistics

To monitor storage consumption, view the usage statistics for Persistent Volume Claims (PVCs). By accessing these metrics, you can track resource use and ensure that your workloads have sufficient capacity.

PVC usage statistics command is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Procedure

To view statistics across a cluster, run the following command:

$ oc adm top pvc -A

Example command output

NAMESPACE     NAME         USAGE(%)
namespace-1   data-etcd-1  3.82%
namespace-1   data-etcd-0  3.81%
namespace-1   data-etcd-2  3.81%
namespace-2   mypvc-fs-gp3 0.00%
default       mypvc-fs     98.36%

To view PVC usage statistics for a specified namespace, run the following command:
```
$ oc adm top pvc -n <namespace_name>
```
- Where <namespace_name> is the name of the specified namespace.
  Example command output
  NAMESPACE NAME USAGE(%) namespace-1 data-etcd-2 3.81% namespace-1 data-etcd-0 3.81% namespace-1 data-etcd-1 3.82%
  In this example, the specified namespace is namespace-1.
To view usage statistics for a specified PVC and for a specified namespace, run the following command:
```
$ oc adm top pvc <pvc_name> -n <namespace_name>
```
- Where <pvc_name> is the name of specified PVC.
- Where <namespace_name> is the name of the specified namespace.
  Example command output
  NAMESPACE NAME USAGE(%) namespace-1 data-etcd-0 3.81%
  In this example, the specified namespace is namespace-1 and the specified PVC is data-etcd-0.

Reduce pod timeouts by using fsGroup

To reduce pod timeouts when using a storage volume with many files, configure the fsGroup field. By specifying this field, you can manage how file ownership and permissions are applied, preventing delays caused by the default recursive permission changes on large volumes.

This can occur because, by default, {product-title} recursively changes ownership and permissions for the contents of each volume to match the fsGroup specified in the securityContext of the pod when that volume is mounted. For volumes with many files, checking and changing ownership and permissions can be time consuming, slowing pod startup. You can use the fsGroupChangePolicy field inside a securityContext to control the way that {product-title} checks and manages ownership and permissions for a volume.

fsGroupChangePolicy defines behavior for changing ownership and permission of the volume before being exposed inside a pod. This field only applies to volume types that support fsGroup-controlled ownership and permissions. This field has two possible values:

OnRootMismatch: Only change permissions and ownership if permission and ownership of root directory does not match with expected permissions of the volume. This can help shorten the time it takes to change ownership and permission of a volume to reduce pod timeouts.
Always: (Default) Always change permission and ownership of the volume when a volume is mounted.

The fsGroupChangePolicy field has no effect on ephemeral volume types, such as secret, configMap, and emptydir.

You can set fsGroupChangePolicy at either the namespace or pod level.

Checking the pods for mismatch

To ensure workload consistency, check the pods running on MicroShift for mismatches. Identifying these discrepancies helps verify that your running workloads match the expected configuration.

Procedure

List the mount point within the first pod by running the following command:

$ oc get pods -n <pod_name_a> -ojsonpath='{.spec.containers[].volumeMounts[].mountPath}'

Replace <pod_name_a> with the name of the first pod.
Example output
```
/files /var/run/secrets/kubernetes.io/serviceaccount
```

List the mount point within the second pod by running the following command:

$ oc get pods -n <pod_name_b> -ojsonpath='{.spec.containers[].volumeMounts[].mountPath}'

Replace <pod_name_b> with the name of the second pod.
Example output
```
/files /var/run/secrets/kubernetes.io/serviceaccount
```

Check the context and permissions inside the first pod by running the following command:

$ oc rsh <pod_name_a> ls -lZah <pvc_mountpoint>

Replace <pod_name_a> with the name of the first pod.

Replace <pvc_mountpoint> with the mount point within the first pod.

Example output

total 12K
dr-xr-xr-x.   1 root root system_u:object_r:container_file_t:s0:c398,c806   40 Feb 17 13:36 .
dr-xr-xr-x.   1 root root system_u:object_r:container_file_t:s0:c398,c806   40 Feb 17 13:36 ..
[...]

Check the context and permissions inside the second pod by running the following command:

$ oc rsh <pod_name_b> ls -lZah <pvc_mountpoint>

Replace <pod_name_b> with the name of the second pod.

Replace <pvc_mountpoint> with the mount point within the second pod.

Example output

total 12K
dr-xr-xr-x.   1 root root system_u:object_r:container_file_t:s0:c15,c25   40 Feb 17 13:34 .
dr-xr-xr-x.   1 root root system_u:object_r:container_file_t:s0:c15,c25   40 Feb 17 13:34 ..
[...]

Compare both the outputs to check if there is a mismatch of SELinux context.

Updating the pods which have mismatch

To resolve configuration discrepancies, update the SELinux context of the pods that display a mismatch status. This process ensures that your running workloads align with the expected configuration, maintaining consistency across your cluster.

Procedure

When there is a mismatch of the SELinux content, create a new security context constraint (SCC) and assign it to both pods. To create a SCC, see "Creating security context constraints".

Update the SELinux context as shown in the following example:

Example output

 [...]
 securityContext:privileged
      seLinuxOptions:MustRunAs
        level: "s0:cXX,cYY"
  [...]

Verifying pods after resolving a mismatch

To confirm that the mismatch is resolved, verify the security context constraint (SCC) and the SELinux label of the pods. Checking these settings ensures that your workloads are functioning with the correct security configurations.

Procedure

Verify that the same SCC is assigned to the first pod by running the following command:
```
$ oc describe pod <pod_name_a> |grep -i scc
```
- Replace <pod_name_a> with the name of the first pod.
  Example output
  openshift.io/scc: restricted
Verify that the same SCC is assigned to first second pod by running the following command:
```
$ oc describe pod <pod_name_b> |grep -i scc
```
- Replace <pod_name_b> with the name of the second pod.
  Example output
  openshift.io/scc: restricted

Verify that the same SELinux label is applied to first pod by running the following command:

$ oc exec <pod_name_a> -- ls -laZ <pvc_mountpoint>

Replace <pod_name_a> with the name of the first pod.

Replace <pvc_mountpoint> with the mount point within the first pod.

Example output

total 4
drwxrwsrwx. 2 root       1000670000 system_u:object_r:container_file_t:s0:c10,c26 19 Aug 29 18:17 .
dr-xr-xr-x. 1 root       root       system_u:object_r:container_file_t:s0:c10,c26 61 Aug 29 18:16 ..
-rw-rw-rw-. 1 1000670000 1000670000 system_u:object_r:container_file_t:s0:c10,c26 29 Aug 29 18:17 test1
[...]

Verify that the same SELinux label is applied to second pod by running the following command:

$ oc exec <pod_name_b> -- ls -laZ <pvc_mountpoint>

Replace <pod_name_b> with the name of the second pod.

Replace <pvc_mountpoint> with the mount point within the second pod.

Example output

total 4
drwxrwsrwx. 2 root       1000670000 system_u:object_r:container_file_t:s0:c10,c26 19 Aug 29 18:17 .
dr-xr-xr-x. 1 root       root       system_u:object_r:container_file_t:s0:c10,c26 61 Aug 29 18:16 ..
-rw-rw-rw-. 1 1000670000 1000670000 system_u:object_r:container_file_t:s0:c10,c26 29 Aug 29 18:17 test1
[...]

Expand persistent volumes

You can expand persistent volumes on MicroShift to increase storage capacity for running workloads without interrupting access to the data. MicroShift supports expansion of CSI volumes, local volumes, and filesystem-backed PVCs through LVMS.

Expanding CSI volumes

You can use the Container Storage Interface (CSI) to expand storage volumes after they have already been created.

Shrinking persistent volumes (PVs) is not supported.

Prerequisites

The underlying CSI driver supports resize.

For information about which CSI drivers support resizing, see under the Additional resources section "CSI drivers supported by {product-title}".
Dynamic provisioning is used.
The controlling StorageClass object has allowVolumeExpansion set to true.

For more information, see section Enabling volume expansion support.

Procedure

For the persistent volume claim (PVC), set .spec.resources.requests.storage to the desired new size.

Verification

To confirm that the resize is finished, look at the status.conditions field of the PVC . {product-title} adds the Resizing condition to the PVC during expansion, which is removed after expansion completes.

Expanding local volumes

To expand your Local Storage Operator (LSO) storage capacity and meet growing data needs, update the storage request in your persistent volume (PV) and persistent volume claim (PVC). This increases capacity for existing volumes without recreating them.

Procedure

Expand the underlying devices. Ensure that appropriate capacity is available on these devices.
Update the corresponding PV objects to match the new device sizes by editing the .spec.capacity field of the PV.
For the storage class that is used for binding the PVC to PV, set the allowVolumeExpansion field to true.
For the PVC, set .spec.resources.requests.storage to match the new size.

Result

Kubelet should automatically expand the underlying file system on the volume, if necessary, and update the status field of the PVC to reflect the new size.

Expanding persistent volume claims (PVCs) with a file system

To expand your storage capacity and meet growing data needs, you can resize existing volumes without recreating them.

Expanding persistent volume claims (PVCs) based on volume types that need file system resizing, such as Google Cloud Platform (GCP) persistent disk (PD), AWS Elastic Block Storage (EBS), and Cinder, is a two-step process. First, expand the volume objects in the cloud provider. Second, expand the file system on the node.

Expanding the file system on the node only happens when a new pod is started with the volume.

Prerequisites

The controlling storage class has the allowVolumeExpansion field set to true.

For more information, see section Enabling volume expansion support.

Procedure

Edit the PVC and request a new size by editing spec.resources.requests. For example, the following expands the ebs PVC to 8 Gi:
Example PVC YAML file
```
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: ebs
spec:
  storageClass: "storageClassWithFlagSet"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
```
Where updating spec.resources.requests to a larger amount expands the PVC.

Verification

After the cloud provider object has finished resizing, the PVC is set to FileSystemResizePending.

Check the condition by running the following command:
```
$ oc describe pvc <pvc_name>
```

Next steps

When the cloud provider object has finished resizing, the PersistentVolume object reflects the newly requested size in PersistentVolume.Spec.Capacity. You can now create or recreate a new pod from the PVC to finish the file system resizing. After the pod is running, the newly requested size is available and the FileSystemResizePending condition is removed from the PVC.

Recovering from failure when expanding volumes

If a resize request fails or remains in a pending state, you can try again by entering a different resize value in .spec.resources.requests.storage for the persistent volume claim (PVC). The new value must be larger than the original volume size.

If entering another smaller resize value in .spec.resources.requests.storage for the PVC does not work, use the following procedure to recover.

Procedure

Mark the persistent volume (PV) that is bound to the PVC with the Retain reclaim policy. Change the persistentVolumeReclaimPolicy field to Retain.
Delete the PVC.
Manually edit the PV and delete the claimRef entry from the PV specification to ensure that the newly created PVC can bind to the PV marked Retain. This marks the PV as Available.
Recreate the PVC in a smaller size, or a size that can be allocated by the underlying storage provider.
Set the volumeName field of the PVC to the name of the PV. This binds the PVC to the provisioned PV only.
Restore the reclaim policy on the PV.

Use ephemeral storage

MicroShift supports ephemeral storage for workloads that require temporary, pod-local data storage. Ephemeral storage is tied to the lifecycle of the pod and is automatically released when the pod terminates. You can configure ephemeral storage limits and use generic ephemeral volumes backed by LVMS for workloads that need more flexible temporary storage.

Overview of ephemeral storage

Use ephemeral storage to provide temporary local storage for stateless applications that only need data for the duration of the pod lifecycle, such as caches, scratch files, and logs that do not need to persist after the pod terminates.

Both developers and administrators can use this feature.

Pods and containers can require ephemeral or transient local storage for their operation. The lifetime of this ephemeral storage does not extend beyond the life of the individual pod, and this ephemeral storage cannot be shared across pods.

Issues related to the lack of local storage accounting and isolation include the following:

Pods cannot detect how much local storage is available to them.
Pods cannot request guaranteed local storage.
Local storage is a best-effort resource.
Pods can be evicted due to other pods filling the local storage, after which new pods are not admitted until sufficient storage is reclaimed.

Unlike persistent volumes, ephemeral storage is unstructured and the space is shared between all pods running on a node, in addition to other uses by the system, the container runtime, and {product-title}. The ephemeral storage framework allows pods to specify their transient local storage needs. It also allows {product-title} to schedule pods where appropriate, and to protect the node against excessive use of local storage.

While the ephemeral storage framework allows administrators and developers to better manage local storage, I/O throughput and latency are not directly affected.

Types of ephemeral storage

Provision ephemeral local storage by creating the primary partition using either root or runtime methods. Choose the method that aligns with your node configuration to ensure temporary storage is available for your workloads.

Root: This partition holds the kubelet root directory, /var/lib/kubelet/ by default, and /var/log/ directory. This partition can be shared between user pods, the operating system, and Kubernetes system daemons. This partition can be consumed by pods through EmptyDir volumes, container logs, image layers, and container-writable layers. Kubelet manages shared access and isolation of this partition. This partition is ephemeral, and applications cannot expect any performance SLAs, such as disk IOPS, from this partition.
Runtime: This is an optional partition that runtimes can use for overlay file systems. {product-title} attempts to identify and provide shared access along with isolation to this partition. Container image layers and writable layers are stored here. If the runtime partition exists, the root partition does not hold any image layer or other writable storage.

Ephemeral storage management overview

Cluster administrators can manage ephemeral storage within a project by setting quotas that define limit ranges and request counts for all pods in a non-terminal state. Developers can also set requests and limits on this resource at the pod and container level.

You can manage local ephemeral storage by specifying requests and limits. Each container in a pod can specify the following:

spec.containers[].resources.limits.ephemeral-storage
spec.containers[].resources.requests.ephemeral-storage

Ephemeral storage management limits and requests

Express ephemeral storage limits and requests using byte quantities with suffixes like G, M, K or power-of-two equivalents Gi, Mi, Ki. Pod-level limits aggregate all container limits plus emptyDir volumes, enabling proper scheduling and preventing pods from exhausting node storage.

Limits and requests for ephemeral storage are measured in byte quantities. You can express storage as a plain integer or as a fixed-point number by using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. For example, the following quantities all represent approximately the same value: 128974848, 129e6, 129M, and 123Mi. Pod-level limits aggregate all container limits plus emptyDir volumes, enabling proper scheduling and preventing pods from exhausting node storage.

The suffixes for each byte quantity are case-sensitive. Be sure to use the correct case. Use the case-sensitive "M", such as used in "400M", to set the request at 400 megabytes. Use the case-sensitive "400Mi" to request 400 mebibytes. If you specify "400m" of ephemeral storage, the storage request is only 0.4 bytes.

The following example configuration file shows a pod with two containers:

Each container requests 2GiB of local ephemeral storage.
Each container has a limit of 4GiB of local ephemeral storage.
At the pod level, kubelet works out an overall pod storage limit by adding up the limits of all the containers in that pod.
- In this case, the total storage usage at the pod level is the sum of the disk usage from all containers plus the emptyDir volumes of a pod.
- Therefore, the pod has a request of 4GiB of local ephemeral storage, and a limit of 8GiB of local ephemeral storage.

Example ephemeral storage configuration with quotas and limits

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: app
    image: images.my-company.example/app:v4
    resources:
      requests:
        ephemeral-storage: "2Gi"
      limits:
        ephemeral-storage: "4Gi"
    volumeMounts:
    - name: ephemeral
      mountPath: "/tmp"
  - name: log-aggregator
    image: images.my-company.example/log-aggregator:v6
    resources:
      requests:
        ephemeral-storage: "2Gi"
      limits:
        ephemeral-storage: "4Gi"
    volumeMounts:
    - name: ephemeral
      mountPath: "/tmp"
  volumes:
    - name: ephemeral
      emptyDir: {}

spec.containers.name.resources.requests.ephemeral-storage: Specifies the container request for local ephemeral storage.
spec.containers.name.resources.limits.ephemeral-storage: Specifies the container limit for local ephemeral storage.

Ephemeral storage management configuration affects pod scheduling and eviction

Configure ephemeral storage requests and limits in the pod spec to control how the scheduler places pods on nodes and when kubelet evicts pods that exceed their allocated storage.

First, the scheduler ensures that the sum of the resource requests of the scheduled containers is less than the capacity of the node. In this case, the pod can be assigned to a node only if the node’s available ephemeral storage (allocatable resource) is more than 4GiB.
Second, at the container level, because the first container sets a resource limit, kubelet eviction manager measures the disk usage of this container and evicts the pod if the storage usage of the container exceeds its limit (4GiB). The kubelet eviction manager also marks the pod for eviction if the total usage exceeds the overall pod storage limit (8GiB).

Overview of generic ephemeral volumes

Generic ephemeral volumes support network-attached storage, size limits, initial data population, and operations like cloning and snapshotting for temporary storage, with some driver-specific limitations.

Generic ephemeral volumes have the following features:

Storage can be local or network-attached.
Volumes can have a fixed size that pods cannot exceed.
Volumes might have some initial data, depending on the driver and parameters.
Typical operations on volumes are supported, assuming that the driver supports them, including snapshotting, cloning, resizing, and storage capacity tracking.

Generic ephemeral volumes do not support offline snapshotting and resizing.

Due to this limitation, the following Container Storage Interface (CSI) drivers do not support the following features for generic ephemeral volumes:

Azure Disk CSI driver does not support resize.
Cinder CSI driver does not support snapshot.

Lifecycle and persistent volume claims

Generic ephemeral volumes follow pod lifecycle through automatically managed persistent volume claims created at pod startup and deleted at termination. Choose volume binding mode and reclaim policy based on this lifecycle behavior.

Generic ephemeral volumes are specified inline in the pod spec and follow the pod’s lifecycle. They are created and deleted along with the pod.

The parameters for a volume claim are allowed inside a volume source of a pod. Labels, annotations, and the whole set of fields for PVCs are supported. When such a pod is created, the ephemeral volume controller then creates an actual PVC object (from the template shown in the Creating generic ephemeral volumes procedure) in the same namespace as the pod, and ensures that the PVC is deleted when the pod is deleted.

This triggers volume binding and provisioning in one of two ways:

Either immediately, if the storage class uses immediate volume binding.

With immediate binding, the scheduler is forced to select a node that has access to the volume after it is available.
When the pod is tentatively scheduled onto a node (WaitForFirstConsumer volume binding mode).

This volume binding option is recommended for generic ephemeral volumes because then the scheduler can choose a suitable node for the pod.

In terms of resource ownership, a pod that has generic ephemeral storage is the owner of the PVCs that provide that ephemeral storage. When the pod is deleted, the Kubernetes garbage collector deletes the PVC, which then usually triggers deletion of the volume because the default reclaim policy of storage classes is to delete volumes. You can create quasi-ephemeral local storage by using a storage class with a reclaim policy of retain. The storage outlives the pod, and in this case, you must ensure that volume clean-up happens separately. While these PVCs exist, they can be used like any other PVC. In particular, they can be referenced as data sources in volume cloning or snapshotting. The PVC object also holds the current status of the volume.

Security

Generic ephemeral volumes allow users who can create pods to indirectly create persistent volume claims (PVCs), even without direct PVC creation permissions. You can restrict this behavior if it conflicts with your security model.

To restrict this behavior, use an admission webhook that rejects objects such as pods that have a generic ephemeral volume.

The normal namespace quota for PVCs still applies, so even if users are allowed to use this new mechanism, they cannot use it to circumvent other policies.

Persistent volume claim naming

Automatically created persistent volume claims (PVCs) are named using pod name and volume name with a hyphen separator, potentially causing conflicts with other pods or manual PVCs.

For example, pod-a with volume scratch and pod with volume a-scratch both end up with the same PVC name, pod-a-scratch.

Such conflicts are detected, and a PVC is only used for an ephemeral volume if it was created for the pod. This check is based on the ownership relationship. An existing PVC is not overwritten or modified, but this does not resolve the conflict. Without the right PVC, a pod cannot start.

Be careful when naming pods and volumes inside the same namespace so that naming conflicts do not occur.

Creating generic ephemeral volumes

To create ephemeral volumes that are automatically provisioned and deleted with pod lifecycle, define a volumeClaimTemplate in your pod spec specifying storage class, size, and access modes.

Procedure

Create the pod object definition and save it to a file.

Include the generic ephemeral volume information in the file.

my-example-pod-with-generic-vols.yaml

kind: Pod
apiVersion: v1
metadata:
  name: my-app
spec:
  containers:
    - name: my-frontend
      image: busybox:1.28
      volumeMounts:
      - mountPath: "/mnt/storage"
        name: data
      command: [ "sleep", "1000000" ]
  volumes:
    - name: data
      ephemeral:
        volumeClaimTemplate:
          metadata:
            labels:
              type: my-app-ephvol
          spec:
            accessModes: [ "ReadWriteOnce" ]
            storageClassName: "gp2-csi"
            resources:
              requests:
                storage: 1Gi

Where spec.volumes.name is the name of the generic ephemeral volume.

Work with volume snapshots

Volume snapshots on MicroShift let you capture the state of a persistent volume at a point in time. You can use snapshots to back up application data, restore volumes to a previous state, and clone volumes for testing or migration. MicroShift uses LVMS-backed CSI snapshot support to create and manage volume snapshot resources.

About volume snapshots

You can use volume snapshots with logical volume manager (LVM) thin volumes to help protect against data loss from applications running in a MicroShift node. MicroShift only supports the logical volume manager storage (LVMS) Container Storage Interface (CSI) provider.

LVMS only supports the volumeBindingMode of the storage class being set to WaitForFirstConsumer. This setting means the storage volume is not provisioned until a pod is ready to mount it.

Example workload that deploys a single pod and PVC

$ oc apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-claim-thin
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: topolvm-provisioner-thin
---
apiVersion: v1
kind: Pod
metadata:
  name: base
spec:
  containers:
  - command:
	    - nginx
	    - -g
	    - 'daemon off;'
    image: registry.redhat.io/rhel8/nginx-122@sha256:908ebb0dec0d669caaf4145a8a21e04fdf9ebffbba5fd4562ce5ab388bf41ab2
    name: test-container
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
    volumeMounts:
    - mountPath: /vol
      name: test-vol
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - name: test-vol
    persistentVolumeClaim:
      claimName: test-claim-thin
EOF

Volume snapshot classes

To enable dynamic snapshotting in LVMS, ensure that at least one VolumeSnapshotClass configuration file is present on the node. This resource defines the Container Storage Interface (CSI) parameters required to create and manage volume snapshots.

You must enable thin logical volumes to take logical volume snapshots.

Example VolumeSnapshotClass configuration file

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: topolvm-snapclass
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "true"
driver: topolvm.io
deletionPolicy: Delete

where:

snapshot.storage.kubernetes.io/is-default-class: Specifies the VolumeSnapshotClass configuration file to use when none is specified by VolumeSnapshot. Where VolumeSnapshot is a request for snapshot of a volume by a user.
driver: Identifies the snapshot provisioner that manages the requests for snapshots of a volume by a user for this class.
deletionPolicy: Specifies the VolumeSnapshotContent objects and the backing snapshots that are kept or deleted when a bound VolumeSnapshot is deleted. Valid values are Retain or Delete.

Creating a volume snapshot

To preserve the data on a PersistentVolumeClaim (PVC) at a specific point in time, create a volume snapshot. By using a volume snapshot, you can restore the volume to its previous state or provision new volumes with the saved data.

To create a snapshot of a MicroShift storage volume, you must first configure RHEL for Edge and the node.

In the following example procedure, the pod that the source volume is mounted to is deleted. Deleting the pod prevents data from being written to it during snapshot creation. Ensuring that no data is being written during a snapshot is crucial to creating a viable snapshot.

Prerequisites

User has root access to a MicroShift node.
MicroShift is running.
A device class defines an LVM thin-pool.
A volumeSnapshotClass specifies driver: topolvm.io.
Any workload attached to the source PVC is paused or deleted. This helps avoid data corruption.

All writes to the volume must be halted while you are creating the snapshot. If you do not halt writes, your data might be corrupted.

Procedure

Prevent data from being written to the volume during snapshotting by using one of the two following steps:
1. Delete the pod to ensure that no data is written to the volume during snapshotting by running the following command:
  $ oc delete my-pod
2. Scale the replica count to zero on a pod that is managed with a replication controller. Setting the count to zero prevents the instant creation of a new pod when one is deleted.
After all writes to the volume are halted, run a command similar to the example that follows. Insert your own configuration details.
Example snapshot configuration
```
# oc apply -f <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: <snapshot_name>
spec:
  volumeSnapshotClassName: topolvm-snapclass
  source:
    persistentVolumeClaimName: test-claim-thin
EOF
```
where:

kind

Specifies the type of VolumeSnapshot object to create.

metadata.name

Specifies the name that you specify for the snapshot.

volumeSnapshotClassName

Specifies the desired name of the VolumeSnapshotClass object.

persistentVolumeClaimName

Specifies either persistentVolumeClaimName or volumeSnapshotContentName. In this example, a snapshot is created from a PVC named test-claim-thin.
Wait for the storage driver to finish creating the snapshot by running the following command:
```
$ oc wait volumesnapshot/<snapshot_name> --for=jsonpath\='{.status.readyToUse}=true'
```
When the volumeSnapshot object is in a ReadyToUse state, you can restore the state as a volume for future PVCs. Restart the pod or scale the replica count back up to the desired number.
After you have created the volume snapshot, you can remount the source PVC to a new pod.

Volume snapshots are located on the same devices as the original data. To use the volume snapshots as backups, move the snapshots to a secure location.

Backing up a volume snapshot

Snapshots of data from applications running on a MicroShift node are created as read-only logical volumes (LVs) located on the same devices as the original data. You must manually mount local volumes before they can be copied as persistent volumes (PVs) and used as backup copies. To use a snapshot of a MicroShift storage volume as a backup, find it on the local host and then move it to a secure location.

Prerequisites

You have root access to the host machine.
You have an existing volume snapshot.

Procedure

Get the name of the volume snapshot by running the following command:
```
$ oc get volumesnapshot -n <namespace> <snapshot_name> -o 'jsonpath={.status.volumeSnapshotContentName}'
```
- Replace <namespace> and <snapshot_name> with the namespace and snapshot name you used.
Get the unique identity of the volume created on the storage backend by using the following command and inserting the name retrieved in the previous step:
```
$ oc get volumesnapshotcontent snapcontent-<retrieved_volume_identity> -o 'jsonpath={.status.snapshotHandle}'
```
- Replace <retrieved_volume_identity> with the volume identity.

Display the snapshots by using the unique identity of the volume you retrieved in the previous step to determine which one you want to backup by running the following command:

$ sudo lvdisplay <retrieved_volume_identity>

Replace <retrieved_volume_identity> with the volume identity.

Example output

--- Logical volume ---
LV Path                /dev/rhel/732e45ff-f220-49ce-859e-87ccca26b14c
LV Name                732e45ff-f220-49ce-859e-87ccca26b14c
VG Name                rhel
LV UUID                6Ojwc0-YTfp-nKJ3-F9FO-PvMR-Ic7b-LzNGSx
LV Write Access        read only
LV Creation host, time rhel-92.lab.local, 2023-08-07 14:45:26 -0500
LV Pool name           thinpool
LV Thin origin name    a2d2dcdc-747e-4572-8c83-56cd873d3b07
LV Status              available
# open                 0
LV Size                1.00 GiB
Mapped size            1.04%
Current LE             256
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           253:11

Create a directory to use for mounting the LV by running the following command:
```
$ sudo mkdir /mnt/snapshot
```
Mount the LV using the device name for the retrieved snapshot handle by running the following command:
```
$ sudo mount /dev/<retrieved_snapshot_handle> /mnt/snapshot
```
- Replace <retrieved_snapshot_handle> with the device name.
Copy the files from the mounted location and store them in a secure location by running the following command:
```
$ sudo cp -r /mnt/snapshot <destination>
```
- Replace <destination> with the path to the secure location.

Restoring a volume snapshot

To recover data from a point-in-time copy, restore a volume snapshot to a new PersistentVolumeClaim (PVC). This process ensures that data from the source volume is preserved and you can verify the integrity of the restored content on the new claim.

The following workflow demonstrates snapshot restoration. In this example, the verification steps are also given to ensure that data written to a source persistent volume claim (PVC) is preserved and restored on a new PVC.

A snapshot must be restored to a PVC of exactly the same size as the source volume of the snapshot. You can resize the PVC after the snapshot is restored successfully if a larger PVC is needed.

Procedure

Restore a snapshot by specifying the VolumeSnapshot object as the data source in a persistent volume claim by entering the following command:

$ oc apply -f <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: snapshot-restore
spec:
  accessModes:
  - ReadWriteOnce
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: my-snap
  resources:
    requests:
      storage: 1Gi
  storageClassName: topolvm-provisioner-thin
---
apiVersion: v1
kind: Pod
metadata:
  name: base
spec:
  containers:
  - command:
      - nginx
	    - -g
	    - 'daemon off;'
    image: registry.redhat.io/rhel8/nginx-122@sha256:908ebb0dec0d669caaf4145a8a21e04fdf9ebffbba5fd4562ce5ab388bf41ab2
    name: test-container
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
    volumeMounts:
    - mountPath: /vol
      name: test-vol
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - name: test-vol
    persistentVolumeClaim:
      claimName: snapshot-restore
EOF

Verification

Wait for the pod to reach the Ready state:

$ oc wait --for=condition=Ready pod/base

When the new pod is ready, verify that the data from your application is correct in the snapshot.

Deleting a volume snapshot

To clean up unneeded snapshots and free storage resources, delete volume snapshots by setting a deletion policy that controls whether the underlying content is retained or removed.

Procedure

Specify the deletion policy that you require in the VolumeSnapshotClass object, as shown in the following example:

Example volumesnapshotclass.yaml file

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-hostpath-snap
driver: hostpath.csi.k8s.io
deletionPolicy: Delete
# ...

When deleting the volume snapshot, if deletionPolicy is set to Delete, the underlying snapshot is deleted along with the VolumeSnapshotContent object. If the Retain value is set, both the underlying snapshot and VolumeSnapshotContent object remain.

If the Retain value is set and the VolumeSnapshot object is deleted without deleting the corresponding VolumeSnapshotContent object, the content remains. The snapshot itself is also retained in the storage back end.

Delete the volume snapshot by entering the following command:
```
$ oc delete volumesnapshot <volumesnapshot_name>
```
Replace <volumesnapshot_name> with the name of the volume snapshot you want to delete.
Example output
```
volumesnapshot.snapshot.storage.k8s.io "mysnapshot" deleted
```
If the deletion policy is set to Retain, delete the volume snapshot content by entering the following command:
```
$ oc delete volumesnapshotcontent <volumesnapshotcontent_name>
```
Replace <volumesnapshotcontent_name> with the content you want to delete.

Optional: If the VolumeSnapshot object is not successfully deleted, enter the following command to remove any finalizers for the leftover resource so that the delete operation can continue:

Only remove the finalizers if you are confident that there are no existing references from either persistent volume claims or volume snapshot contents to the VolumeSnapshot object. Even with the --force option, the delete operation does not delete snapshot objects until all finalizers are removed.

$ oc patch -n $PROJECT volumesnapshot/$NAME --type=merge -p '{"metadata": {"finalizers":null}}'

Example output

volumesnapshotclass.snapshot.storage.k8s.io "csi-ocs-rbd-snapclass" deleted

The finalizers are removed and the volume snapshot is deleted.

About LVM volume cloning

You can use the logical volume manager storage (LVMS) for persistent volume claim (PVC) cloning of the logical volume manager (LVM) thin volumes. A clone is a duplicate of an existing volume that can be used like any other volume.

When you provision the clone, an exact duplicate of the original volume is created if the data source references a source PVC in the same namespace. After a cloned PVC is created, the cloned VPC is considered a new object and completely separate from the source PVC. The clone represents a snapshot of the data from the source at the moment in time.

Cloning is only possible when the source and destination PVCs are in the same namespace. To create PVC clones, you must configure thin volumes on the RHEL for Edge host.

Disable the LVMS CSI provider and CSI snapshot

If your MicroShift deployment uses external or pre-provisioned storage and does not require dynamic local provisioning, you can disable the LVMS CSI driver and CSI snapshot controller. Disabling these components reduces resource usage on constrained devices.

Disabling and uninstalling LVMS CSI provider and CSI snapshot deployments

To reduce the use of runtime resources, such as RAM, CPU, and storage, remove or disable the LVMS CSI provider and CSI snapshot deployments. This configuration optimizes system performance by eliminating storage components that are not required for your specific workload.

You can configure MicroShift to disable CSI provider and CSI snapshot only before installing and running MicroShift. After MicroShift is installed and running, you must update the configuration file and uninstall the components.

To reduce the use of runtime resources, you can remove or disable the following storage components:

You can configure MicroShift to disable the built-in logical volume manager storage (LVMS) Container Storage Interface (CSI) provider.
You can configure MicroShift to disable the Container Storage Interface (CSI) snapshot capabilities.
You can uninstall the installed CSI implementations using oc commands.

Automated uninstallation is not supported as this can cause orphaning of the provisioned volumes. Without the LVMS CSI driver, the node does not detect the underlying storage interface and cannot perform provisioning and deprovisioning or mounting and unmounting operations.

Disable deployments that run CSI snapshot implementations

To prevent the installation of CSI implementation pods, disable the deployments that run CSI snapshot implementations. This configuration conserves system resources by ensuring that snapshot components are not deployed when they are not required.

Use the procedure if you are defining the configuration file before installing and running MicroShift. If MicroShift is already started, the CSI snapshot implementation will be running. You must manually remove the implementation by following the uninstallation instructions.

MicroShift does not delete CSI snapshot implementation pods. You must configure MicroShift to disable installation of the CSI snapshot implementation pods during the startup process.

Procedure

Disable installation of the CSI snapshot controller by entering the optionalCsiComponents value under the storage section of the MicroShift configuration file in /etc/microshift/config.yaml:
```
# ...
  storage: {}
# ...
```
where:

storage

Specifies the storage details. You can choose to not define optionalCsiComponents. If you do specify the optionalCsiComponents field, valid values include: an empty value ([]) or a single empty string element ([""]), snapshot-controller, or none. A value of none is mutually exclusive with all other values.

If the optionalCsiComponents value is empty or null, MicroShift defaults to deploying snapshot-controller.
After the optionalCsiComponents field is specified with a supported value in the config.yaml, start MicroShift by running the following command:
```
$ sudo systemctl start microshift
```
MicroShift does not redeploy the disabled components after a restart.

Disable deployments that run the CSI driver implementations

You can disable installation of the CSI implementation pods. MicroShift does not delete CSI driver implementation pods. You must configure MicroShift to disable installation of the CSI driver implementation pods during the startup process.

This procedure is for defining the configuration file before installing and running MicroShift. If MicroShift is already started, then the CSI driver implementation is running. You must manually remove it by following the uninstallation instructions.

Procedure

Disable installation of the CSI driver by entering the driver value under the storage section of the MicroShift configuration file in /etc/microshift/config.yaml:
```
# ...
  storage
   driver:
   - "none"
# ...
```
where:

storage.driver.none

Specifies the driver to disable. Valid values are none or lvms.

By default, the driver value is empty or null and LVMS is deployed.
Start MicroShift after the driver field is specified with a supported value in the /etc/microshift/config.yaml file by running the following command:
```
$ sudo systemctl enable --now microshift
```
MicroShift does not redeploy the disabled components after a restart operation.

Uninstalling the CSI snapshot implementation

To remove the Container Storage Interface (CSI) snapshot capability from your cluster, uninstall the CSI snapshot implementation.

Prerequisites

MicroShift is installed and running.
The CSI snapshot implementation is deployed on the MicroShift node.

Procedure

Uninstall the CSI snapshot implementation by running the following command:

$ oc delete -n kube-system deployment.apps/snapshot-controller

Example output

deployment.apps "snapshot-controller" deleted

Uninstalling the CSI driver implementation

To remove the Container Storage Interface (CSI) integration from your cluster, uninstall the CSI driver implementation.

Prerequisites

MicroShift is installed and running.
The CSI driver implementation is deployed on the MicroShift node.

Procedure

Delete the lvmclusters object by running the following command:

$ oc delete -n openshift-storage lvmclusters.lvm.topolvm.io/lvms

Example output

lvmcluster.lvm.topolvm.io "lvms" deleted

Delete the lvms-operator by running the following command:

$ oc delete -n openshift-storage deployment.apps/lvms-operator

Example output

deployment.apps "lvms-operator" deleted

Delete the topolvm-provisioner StorageClass by running the following command:

$ oc delete storageclasses.storage.k8s.io/topolvm-provisioner

Example output

storageclass.storage.k8s.io "topolvm-provisioner" deleted