To deploy multiple Model Serving Runtime in NERC RHOAI

[Milstein]

Can we check and install all these or most of them to our OCP Test and Prod setup:

https://github.com/rh-aiservices-bu/llm-on-openshift/tree/main?tab=readme-ov-file#inference-servers

How to configure is explained here.

[Milstein]

Our current Model Server only offers "OpenVINO" Serving runtime:

[IsaiahStapleton]

@Milstein We do not have only OpenVINO serving runtimes. For some reason though, only that shows up as an option when using a Data Science Project created by kube:admin. These are all of the serving runtimes that come by default with RHOAI, except for LlamaCPP which I manually added::

[IsaiahStapleton]

Also, our test cluster currently has an issue with KServe. When trying to deploy a model the kserve webhook fails to call.

ai-performance-profiling 6s Warning InternalError inferenceservice/granite fails to reconcile predictor: Internal error occurred: failed calling webhook "webhook.serving.knative.dev": failed to call webhook: Post "https://webhook.knative-serving.svc:443/defaulting?timeout=10s": context deadline

However when inspecting the knative-serving namespace there does not appear to be an issue with the webhook pod so I'm not sure the issue. I was granted access to the Albany cluster and I set up RHOAI from scratch for model serving and have not run into any issues with model serving there, and currently have granite model deployed there.

[Milstein]

Also we need to check if this runtime can be added: https://developers.redhat.com/articles/2024/11/22/how-ramalama-makes-working-ai-models-boring#why_not_just_use_ollama_