Red Hat build of MicroShift

Create a ServingRuntime CR for use in MicroShift

You can create a ServingRuntime custom resource (CR) based on installed manifests and release information.

The included steps are an example of reusing the included microshift-ai-model-serving manifest files to re-create the OpenVINO Model Server (OVMS) model-serving runtime in the workload namespace.

This approach does not require a live node, so it can be part of CI/CD automation.

Prerequisites

Both the microshift-ai-model-serving and microshift-ai-model-serving-release-info RPMs are installed.
You have root user access to your machine.
The OpenShift CLI (oc) is installed.

Procedure

Extract the image reference of the ServingRuntime CR you want to use from the MicroShift release information file by running the following command:
```
$ OVMS_IMAGE="$(jq -r '.images | with_entries(select(.key == "ovms-image")) | .[]' /usr/share/microshift/release/release-ai-model-serving-"$(uname -i)".json)"
```
In this example, the image reference for the OVMS model-serving runtime is extracted.

Copy the original ServingRuntime YAML file by running the following command:

$ cp /usr/lib/microshift/manifests.d/050-microshift-ai-model-serving-runtimes/ovms-kserve.yaml ./ovms-kserve.yaml

Add the actual image reference to the image: parameter field value of the ServingRuntime YAML by running the following command:
```
$ sed -i "s,image: ovms-image,image: ${OVMS_IMAGE}," ./ovms-kserve.yaml
```

Create the ServingRuntime object in a custom namespace using the YAML file by running the following command:

$ oc create -n <ai_demo> -f ./ovms-kserve.yaml

where:

<ai_demo>: Specifies the name of your namespace.

If the ServingRuntime CR is part of a new manifest, set the namespace in the kustomization.yaml file, for example:

Example Kustomize manifest namespace value

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ai-demo
resources:
  - ovms-kserve.yaml
#...

Next steps

Create the InferenceService object.
Verify that your model is ready for inferencing.
Query the model.
Optional: Examine the model metrics.