Scalable ML Service

Quick start:

1. Clone this repo
2. Prepare the docker images

   docker build -t smls-server ./api_server
   docker build -t smls-model-runner ./model_runner
   docker build -t smls-frontend-app ./frontend_app
   

3. docker-compose up
4. Navigate to http://localhost:3000/api/graphiql
5. Start writing french!

The Project: The goal is to build a full architecture to gather sentence pairs in various languages and run the infrastructure to build the machine learned models and provide a translations interface.

The project is purely educational and offers me a change to work with

• Machine learning
• Scalable architectures
• Application development
• Infrastructure development
• API development
• Test development
• Benchmarking a solution
• Etc.

It will not, however, offer a training ground for general DevOps, the art of keeping the uptime near 100%, live migrations, continuous integration/deployment etc.

Related blog articles are:

1. Scalable Machine Learning Service
2. Productizing Machine Learning Models

Further things to do as educational tasks:

• Auto Scaling: Currently we statically set the number of instances we run. ideally this would be adjusted depending on load.
• Asynchronous: Currently the GraphQl interface is synchronous. one makes a query and receives an answer. though for long running tasks this is not optimal and we can push the asynchronous semantics all the way to the client.
• Multiple models: How would we architect for a system that serves several ML models. Maybe a model per user.
• Resilience: Currently stuff like back off, timeouts, etc has not been implemented. This would also be needed.
• Development Setup: Currently it is hard to develop anything as there are strong dependencies.
• Migrations: Currently the Postgres database is dead weight. However, in a real life situation we need to handle migration etc. as a part of the architecture.
• Training: Integrate a training pipeline for models based on changes of data og changes of the model they are trained on.
• Save Predictions: Dave the predictions made to "improve service".

Resources

Thanks to the authors behind following resources.

1. https://www.rabbitmq.com/tutorials/tutorial-six-elixir.html
2. https://www.rabbitmq.com/tutorials/tutorial-six-python.html
3. https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

And thanks to the entire community making projects like these possible.