Model serving with flyte

[aswanthkrishna]

I have been using flyte quite successfully for model training and batch predictions. I was wondering if flyte provides a way to serve ML model as a server in Kubernetes natively?

What would be the best pattern to achieve something like this?

[dosubot[bot]]

🤖

Hey @aswanthkrishna! Great to see you back. Hope you've been doing awesome! 😊

Flyte itself does not natively serve ML models as a server in Kubernetes. However, it does have the capability to integrate with other tools that specialize in model serving, such as TensorFlow Serving, TorchServe, or NVIDIA Triton Inference Server. These tools can be orchestrated by Kubeflow Serving or Seldon Core, which can be integrated into a Kubernetes environment alongside Flyte.

In terms of patterns, you could consider a pattern where Flyte is responsible for the orchestration of your workflows, including the training of your models, and another tool is responsible for serving the trained models. Once your model is trained with Flyte, you could store the model in a location accessible by your model serving tool, and then trigger the serving tool to load and serve the new model.

There is a solved issue in the Flyte repository that discusses the implementation of a plugin for model serving using kserve. This could potentially be a starting point for you to implement a similar plugin for your chosen model serving tool.

Please note that while Flyte has plugins for managing TensorFlow jobs on Kubernetes through the Kubeflow Training Operator, these are focused on training ML models rather than serving them. The code in the context provided is specifically for creating, monitoring, and managing TensorFlow training jobs.

I hope this helps! If you have any further questions, feel free to ask.

Relevant sources:

• flyteplugins/go/tasks/plugins/k8s/kfoperators/tensorflow/tensorflow.go
• [Plugin]Model serving using kserve

Sources

• [Plugin]Model serving using kserve
• flyteplugins/go/tasks/plugins/k8s/kfoperators/tensorflow/tensorflow.go

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[kumare3]

We are working on Flyte serving agent for sagemaker. This is coming soon in couple weeks. And then you can integrate with anything else too.

Also definitely talk to us at Union for more serving info