Supported Red Hat OpenShift AI Self-Managed model-serving runtimes
You can review the following information to learn about the Red Hat OpenShift AI Self-Managed model-serving runtimes that are verified for MicroShift deployments.
The following Red Hat OpenShift AI Self-Managed model-serving runtimes that are verified for MicroShift deployments:
-
vLLM ServingRuntime for KServe
-
OpenVINO Model Server
The OpenVINO Model Server does not support the IPv6 networking protocol. Check each model server before use to ensure that it supports your networking configuration.
The following runtimes are available for development purposes with MicroShift:
-
Caikit Text Generation Inference Server (Caikit-TGIS) ServingRuntime for KServe
-
Caikit Standalone ServingRuntime for KServe
-
Text Generation Inference Server (TGIS) Standalone ServingRuntime for KServe
-
vLLM ServingRuntime with Gaudi accelerators support for KServe
-
vLLM ROCm ServingRuntime for KServe
-
Custom runtimes that you create and test