Serve AI models on MicroShift

You can review the following information to learn how to serve models on the Red Hat OpenShift AI Self-Managed single-model serving platform in MicroShift by configuring a model-serving runtime using the ServingRuntime and InferenceService custom resource (CRs).

Model-serving runtimes for AI models in MicroShift

A model-serving runtime is an environment for deploying and managing an AI model, providing integration with a specified model server and the model frameworks that it supports. Creating the model-serving runtime means configuring the objects that select the right model format for the AI model and serve the queries, among other detailed functions specific your deployment.

The ServingRuntime custom resource

The ServingRuntime CR is a YAML file that defines the templates for pods that can dynamically load and unload AI model formats and exposes a service endpoint for querying the model through the API. Each ServingRuntime CR contains the information needed to run AI models, including the container image of the runtime and a list of the model formats that the model-serving runtime supports. Other configuration settings for the model-serving runtime can be set with environment variables defined in the container specification.

The InferenceService custom resource

The InferenceService CR is a YAML file that creates a server or inference service to process inference queries, pass them to the model, then return the inference output. In MicroShift, the output is returned in the CLI. This inference service configuration file can also include many other options, such as specifying a hardware accelerator.

Important

As a single-node Kubernetes distribution, MicroShift does not support multi-model deployments. You must use the single-model serving platform. In each MicroShift deployment, you can use one AI model, but potentially more than one model runtime.
Workflow for configuring a model-serving runtime

  • Select the model-serving runtime that supports the format of your AI model.

  • Create the ServingRuntime CR in your workload namespace.

  • If the MicroShift node is already running, you can export the required ServingRuntime CR to a file and edit it.

  • If the MicroShift node is not running, or if you want to manually prepare a manifest, you can use the original definition on the disk, which is part of the microshift-ai-model-serving RPM.

  • Create the InferenceService CR in your workload namespace.

← Previous: Package your AI model into an OCI image | ↑ Up: AI | ⌂ Home: Red Hat build of MicroShift | Next: Supported Red Hat OpenShift AI Self-Managed custom resource definitions →