Skip to content

Commit

Permalink
seblum published a site update
Browse files Browse the repository at this point in the history
  • Loading branch information
seblum committed Feb 3, 2024
1 parent ca05a94 commit f1a8746
Show file tree
Hide file tree
Showing 106 changed files with 11,514 additions and 7,854 deletions.
8 changes: 4 additions & 4 deletions 00-Preamble.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ This project started out of an interest in multiple domains. At first, I was wor

The overall aim is to build and create a MLOps architecture based on Airflow running on AWS EKS. Ideally, this architecture is create using terraform. Model tracking might be done using MLFlow, Data tracking using DVC. Further mentioned might be best practices in software development, CI/CD, Docker, and pipelines. I might also include a small Data Science use case utilizing the Airflow Cluster we built.

The book contains two sections with distinct focuses. The first section comprises Chapters 3 to 6, which consist of tutorials on the specific tools aforementioned. These chapters also serve as prerequisites for the subsequent sections. Among these tutorials, the chapters dedicated to *Airflow* and *MLflow* are oriented towards Data Scientists, providing insights into their usage. The chapters centered around *Kubernetes* and *Terraform* target Data- and MLOps Engineers, offering detailed guidance on deploying and managing these tools.
The book contains two parts with distinct focuses. The first part comprises Chapters [3: Airflow](#airflow), [4: MLflow](#mlflow), [5: Kubernetes](#kubernetes), and [6: Terraform](#terraform), which consist of tutorials on the specific tools aforementioned. These chapters also serve as prerequisites for the subsequent part. Among these tutorials, the chapters dedicated to *Airflow* and *MLflow* are oriented towards Data Scientists, providing insights into their usage. The chapters centered around *Kubernetes* and *Terraform* target Data- and MLOps Engineers, offering detailed guidance on deploying and managing these tools.

The second section, comprising Chapters 7 to 9, delves into an exemplary machine learning Platform. This section demands a strong background in engineering due to its complexity. While these chapters cover the essential tools introduced in the previous section, they may not explore certain intricate aspects used like OAuth authentication and networking details in great depth. Moreover, it is crucial to note that the ML Platform example presented is not intended for production deployment, as there should be significant security concerns considered. Instead, its main purpose is to serve as an informative illustration of ML platforms and MLOps engineering principles.
The second part, comprising Chapters [7: ML Platform Design](#ml-platform-design), [8: Platform Deployment](#platform-deployment), and [9: Use Case Development](#use-case-development), delves into an exemplary machine learning Platform. This part of the book demands a strong background in engineering due to its complexity. While these chapters cover the essential tools introduced in the previous part, they may not explore certain intricate aspects used like OAuth authentication and networking details in great depth. Moreover, it is crucial to note that the ML Platform example presented is not intended for production deployment, as there should be significant security concerns considered. Instead, its main purpose is to serve as an informative illustration of ML platforms and MLOps engineering principles.

Chapter 1 and 2 serve as an introduction to the domain of MLOps
Chapter [1: Introduction](#introduction) and [2: Ops Tools and Principles](#ops-tools-and-principles) serve as an introduction to the domain of MLOps

### A work in progress {.unlisted .unnumbered}

This project / book / tutorial / whatever this is or will be, startet by explaining the concept of Kubernetes. The plan is to continuously update it by further sections. Since there is no deadline, there is no timeline, and I am also not sure whether there will exist something final to be honest.
This project / book / tutorial / whatever this is or will be, startet by explaining the concept of Kubernetes. The plan is to continuously update it by further chapters. Since there is no deadline, there is no timeline, and I am also not sure whether there will exist something final to be honest.

This document is written during my journey in the realm of MLOps. It is therefore in a state of continuous development.

6 changes: 5 additions & 1 deletion 01.1-Introduction-Machine_Learning_Workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,16 @@ A machine learning workflow typically involves several stages. These stages are

**1. Data Preparation**
In the first stage, data used to train a machine learning model is collected, cleaned, and preprocessed. Preprocessing includes tasks to remove missing or duplicate data, normalize data, and split data into a training and testing set.

**2. Model Building**
In the second stage, a machine learning model is selected and trained using the prepared data. This includes tasks such as selecting an appropriate algorithm as a machine learning model, training the model, and tuning the model's parameters to improve its performance.

**3. Model Evaluation**
Afterward, the performance of the trained model is evaluated using the test data set. This includes tasks such as measuring the accuracy and other performance metrics, comparing the performance of different models, and identifying potential issues with the model.

**4. Model Deployment**
Finally, the selected and optimized model is deployed to a production environment where it can be used to make predictions on new data. This stage includes tasks like scaling the model to handle large amounts of data, and deploying the model to different environments to be used in different contexts

**5. Model Monitoring and Maintenance**
It is important to monitor the model performance and update the model as needed, once the model is deployed. This includes tasks such as collecting feedback from the model, monitoring the model's performance metrics, and updating the model as necessary.

Expand Down Expand Up @@ -52,7 +56,7 @@ Model Tracking tools are often used at the development and testing stages of the

Model Serving refers to the process of deploying a machine learning model in a production environment, so it can be used to make predictions on new data. This includes tasks such as scaling the model to handle large amounts of data, deploying the model to different environments, and monitoring the performance of the deployed model. Model serving tools are specifically used at the deployment stage of the machine learning workflow and can handle the necessary tasks mentioned beforehand.

There are multiple tools that integrate the funtionality of serving models, each different in its specific use cases, for example *KF Serve*, *BentoML*, *Seldon*, or also the already mentioned tools like *MLflow*.
There are multiple tools that integrate the funtionality of serving models, each different in its specific use cases, for example *KF Serve*, *BentoML*, *Seldon*, *AWS Sagemaker* or also the already mentioned tools like *MLflow*.

#### Integrated development environment

Expand Down
4 changes: 2 additions & 2 deletions 01.2-Introduction-MLOps.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,15 @@ Another key difference is that MLOps places a great emphasis on model governance

1. *Machine Learning Platforms*: Platforms such as TensorFlow, PyTorch, and scikit-learn are used to develop and train machine learning models.
2. *Experiment Management Tools*: Tools such as MLflow, Weights & Biases, and Airflow are used to track, version, and reproduce experiments, as well as to manage the model lifecycle.
3. *Model Deployment and Management Tools*: Tools such as TensorFlow Serving, Clipper, and Seldon Core are used to deploy and manage machine learning models in production.
3. *Model Deployment and Management Tools*: Tools such as TensorFlow Serving, AWS Sagemaker, and Seldon Core are used to deploy and manage machine learning models in production.
4. *Data Versioning and Management Tools*: Tools such as DVC (Data Version Control) and Pachyderm are used to version and manage the data used for training and evaluating models.
5. *Automated Model Evaluation and Deployment Tools*: Tools such as AlgoTrader and AlgoHub are used to automate the evaluation of machine learning models and deploy the best-performing models to production.

It's important to note that the specific tools used in MLOps and DevOps may vary depending on the organization's needs. Some of the tools are used in both but applied differently in each, e.g. container orchestration tools like Kubernetes. The above lists do also not claim to be complete and there are of course multiple more tools.

### MLOps Lifecycle

Incorporating the tools introduced by DevOps and MLOps can extend the machine learning workflow outlined in the previous section, resulting in a complete MLOps lifecycle that covers each stage of the machine learning process while integrating automation practices.
Incorporating the tools introduced by DevOps and MLOps can extend the machine learning workflow outlined in the [previous section](#machine-learning-workflow), resulting in a complete MLOps lifecycle that covers each stage of the machine learning process while integrating automation practices.

![](images/01-Introduction/mlops-lifecycle.svg){ width=100% }

Expand Down
Loading

0 comments on commit f1a8746

Please sign in to comment.