Understanding the verification of container images lacking verifiable signatures

Each Red Hat OpenShift Container Platform release image is immutable and signed with a Red Hat production key. During cluster update or installation, a release image might deploy container images without a verifiable signature. The signature on the release image validates all release contents transitively.

For example, the image references lacking a verifiable signature are contained in the signed Red Hat OpenShift Container Platform release image:

Example release info output
$ oc adm release info quay.io/openshift-release-dev/ocp-release@sha256:2309578b68c5666dad62aed696f1f9d778ae1a089ee461060ba7b9514b7ca417 -o pullspec
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aafb914d5d7d0dec4edd800d02f811d7383a7d49e500af548eab5d00c1bffdb

The first line specifies the signed release image SHA. The second line specifies a container image lacking a verifiable signature that is included in the release.

Automated verification during updates

Verification of signatures is automatic. The OpenShift Cluster Version Operator (CVO) verifies signatures on the release images during an Red Hat OpenShift Container Platform update. This is an internal process. An Red Hat OpenShift Container Platform installation or update fails if the automated verification fails.

Verification of signatures can also be done manually using the skopeo command-line utility.

Using skopeo to verify signatures of Red Hat container images

You can verify the signatures for container images included in an Red Hat OpenShift Container Platform release image by pulling those signatures from the Red Hat OpenShift Container Platform release mirror site.

Because the signatures on the mirror site are not in a format readily understood by Podman or CRI-O, you can use the skopeo standalone-verify command to verify that your release images are signed by Red Hat.

Prerequisites

  • You have installed the skopeo command-line utility.

Procedure

  1. Get the full SHA for your release by running the following command:

    $ oc adm release info <release_version>

    • Substitute <release_version> with your release number, for example, 4.14.3.

      Example output snippet
      ---
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:e73ab4b33a9c3ff00c9f800a38d69853ca0c4dfa5a88e3df331f66df8f18ec55
---

  2. Pull down the Red Hat release key by running the following command:

    $ curl -o pub.key https://access.redhat.com/security/data/fd431d51.txt

  3. Get the signature file for the specific release that you want to verify by running the following command:

    $ curl -o signature-1 https://mirror.openshift.com/pub/openshift-v4/signatures/openshift-release-dev/ocp-release/sha256=<sha_from_version>/signature-1

    Replace <sha_from_version> with SHA value from the full link to the mirror site that matches the SHA of your release. For example, the link to the signature for the 4.12.23 release is https://mirror.openshift.com/pub/openshift-v4/signatures/openshift-release-dev/ocp-release/sha256=e73ab4b33a9c3ff00c9f800a38d69853ca0c4dfa5a88e3df331f66df8f18ec55/signature-1, and the SHA value is e73ab4b33a9c3ff00c9f800a38d69853ca0c4dfa5a88e3df331f66df8f18ec55.

  4. Get the manifest for the release image by running the following command:

    $ skopeo inspect --raw docker://<quay_link_to_release> > manifest.json

    Replace <quay_link_to_release> with the output of the oc adm release info command. For example, quay.io/openshift-release-dev/ocp-release@sha256:e73ab4b33a9c3ff00c9f800a38d69853ca0c4dfa5a88e3df331f66df8f18ec55.

  5. Use skopeo to verify the signature:

    $ skopeo standalone-verify manifest.json quay.io/openshift-release-dev/ocp-release:<release_number>-<arch> any signature-1 --public-key-file pub.key

    where:

    <release_number>

    Specifies the release number, for example 4.14.3.

    <arch>

    Specifies the architecture, for example x86_64.

    Example output
    Signature verified using fingerprint 567E347AD0044ADE55BA8A5F199E2F91FD431D51, digest sha256:e73ab4b33a9c3ff00c9f800a38d69853ca0c4dfa5a88e3df331f66df8f18ec55

← Previous: Verifying the signature verification configuration | ↑ Up: Control image deployment using trusted sources and signature verification | ⌂ Home: Red Hat OpenShift Container Platform | Next: About sigstore →