feat: add final deployment (#16)

* add initial structure to deployment and usage

* update repository depiction and k8s dag

* added updated code

* add updated train_model function

* add new pipeline image

* add updated model comparison code

* update and add deploy_model_to_sagemaker

* add inferencing code

* add user profiles to terraform code

* extend glossary list

* update airflow deployment code

* update mlflow description

* list airflow helm components

* add jupyter helm description list

* add monitoring content

* add Dashboard configuration explanation

* add sagemaker tf description

* add Airflow final example

* add link to airflow final example

* Revert "add Airflow final example"

This reverts commit 1926e89.

* add airflow final example files

* change directory names of files

* add installation paragraph

* final terraform example

* add code to airflow final example

* add contribution chapter

* remove seldon mentions

* add subsections to Platform design

* split chapter 2 into different files

* Add Github Actions paragraph

* rename subsection

* fix helm description

* make python operator beautiful

* fix markdown links

* rename subsection

* add prerequisites command

* resolve TODOs

* add links to chapters and sections

* add statefulset

* distinguish health checks

* Note for Transfer Operators

* feat: minor fixed

manuscript/.gitignore

@@ -13,7 +13,7 @@ Rmd_book_template
 
 .DS_Store
 files/.DS_Store
-files/02-Terraform/.DS_Store
+files/06-Terraform/.DS_Store
 
 # MLflow runs folder
 **/mlruns

manuscript/00-Preamble.Rmd

@@ -6,15 +6,15 @@ This project started out of an interest in multiple domains. At first, I was wor
 
 The overall aim is to build and create a MLOps architecture based on Airflow running on AWS EKS. Ideally, this architecture is create using terraform. Model tracking might be done using MLFlow, Data tracking using DVC. Further mentioned might be best practices in software development, CI/CD, Docker, and pipelines. I might also include a small Data Science use case utilizing the Airflow Cluster we built.
 
-The book contains two sections with distinct focuses. The first section comprises Chapters 3 to 6, which consist of tutorials on the specific tools aforementioned. These chapters also serve as prerequisites for the subsequent sections. Among these tutorials, the chapters dedicated to *Airflow* and *MLflow* are oriented towards Data Scientists, providing insights into their usage. The chapters centered around *Kubernetes* and *Terraform* target Data- and MLOps Engineers, offering detailed guidance on deploying and managing these tools.
+The book contains two parts with distinct focuses. The first part comprises Chapters [3: Airflow](#airflow), [4: MLflow](#mlflow), [5: Kubernetes](#kubernetes), and [6: Terraform](#terraform), which consist of tutorials on the specific tools aforementioned. These chapters also serve as prerequisites for the subsequent part. Among these tutorials, the chapters dedicated to *Airflow* and *MLflow* are oriented towards Data Scientists, providing insights into their usage. The chapters centered around *Kubernetes* and *Terraform* target Data- and MLOps Engineers, offering detailed guidance on deploying and managing these tools.
 
-The second section, comprising Chapters 7 to 9, delves into an exemplary machine learning Platform. This section demands a strong background in engineering due to its complexity. While these chapters cover the essential tools introduced in the previous section, they may not explore certain intricate aspects used like OAuth authentication and networking details in great depth. Moreover, it is crucial to note that the ML Platform example presented is not intended for production deployment, as there should be significant security concerns considered. Instead, its main purpose is to serve as an informative illustration of ML platforms and MLOps engineering principles.
+The second part, comprising Chapters [7: ML Platform Design](#ml-platform-design), [8: Platform Deployment](#platform-deployment), and [9: Use Case Development](#use-case-development), delves into an exemplary machine learning Platform. This part of the book demands a strong background in engineering due to its complexity. While these chapters cover the essential tools introduced in the previous part, they may not explore certain intricate aspects used like OAuth authentication and networking details in great depth. Moreover, it is crucial to note that the ML Platform example presented is not intended for production deployment, as there should be significant security concerns considered. Instead, its main purpose is to serve as an informative illustration of ML platforms and MLOps engineering principles.
 
-Chapter 1 and 2 serve as an introduction to the domain of MLOps
+Chapter [1: Introduction](#introduction) and [2: Ops Tools and Principles](#ops-tools-and-principles) serve as an introduction to the domain of MLOps
 
 ### A work in progress {.unlisted .unnumbered}
 
-This project / book / tutorial / whatever this is or will be, startet by explaining the concept of Kubernetes. The plan is to continuously update it by further sections. Since there is no deadline, there is no timeline, and I am also not sure whether there will exist something final to be honest.
+This project / book / tutorial / whatever this is or will be, startet by explaining the concept of Kubernetes. The plan is to continuously update it by further chapters. Since there is no deadline, there is no timeline, and I am also not sure whether there will exist something final to be honest.
 
 This document is written during my journey in the realm of MLOps. It is therefore in a state of continuous development. 
 

manuscript/01.1-Introduction-Machine_Learning_Workflow.Rmd

@@ -7,12 +7,16 @@ A machine learning workflow typically involves several stages. These stages are
 
 **1. Data Preparation**
 In the first stage, data used to train a machine learning model is collected, cleaned, and preprocessed. Preprocessing includes tasks to remove missing or duplicate data, normalize data, and split data into a training and testing set.
+
 **2. Model Building** 
 In the second stage, a machine learning model is selected and trained using the prepared data. This includes tasks such as selecting an appropriate algorithm as a machine learning model, training the model, and tuning the model's parameters to improve its performance.
+
 **3. Model Evaluation**
 Afterward, the performance of the trained model is evaluated using the test data set. This includes tasks such as measuring the accuracy and other performance metrics, comparing the performance of different models, and identifying potential issues with the model.
+
 **4. Model Deployment** 
 Finally, the selected and optimized model is deployed to a production environment where it can be used to make predictions on new data. This stage includes tasks like scaling the model to handle large amounts of data, and deploying the model to different environments to be used in different contexts
+
 **5. Model Monitoring and Maintenance**
 It is important to monitor the model performance and update the model as needed, once the model is deployed. This includes tasks such as collecting feedback from the model, monitoring the model's performance metrics, and updating the model as necessary.
 
@@ -52,7 +56,7 @@ Model Tracking tools are often used at the development and testing stages of the
 
 Model Serving refers to the process of deploying a machine learning model in a production environment, so it can be used to make predictions on new data. This includes tasks such as scaling the model to handle large amounts of data, deploying the model to different environments, and monitoring the performance of the deployed model. Model serving tools are specifically used at the deployment stage of the machine learning workflow and can handle the necessary tasks mentioned beforehand.
 
-There are multiple tools that integrate the funtionality of serving models, each different in its specific use cases, for example *KF Serve*, *BentoML*, *Seldon*, or also the already mentioned tools like *MLflow*.
+There are multiple tools that integrate the funtionality of serving models, each different in its specific use cases, for example *KF Serve*, *BentoML*, *Seldon*, *AWS Sagemaker* or also the already mentioned tools like *MLflow*.
 
 #### Integrated development environment
 

manuscript/01.2-Introduction-MLOps.Rmd

@@ -32,15 +32,15 @@ Another key difference is that MLOps places a great emphasis on model governance
 
 1.  *Machine Learning Platforms*: Platforms such as TensorFlow, PyTorch, and scikit-learn are used to develop and train machine learning models.
 2.  *Experiment Management Tools*: Tools such as MLflow, Weights & Biases, and Airflow are used to track, version, and reproduce experiments, as well as to manage the model lifecycle.
-3.  *Model Deployment and Management Tools*: Tools such as TensorFlow Serving, Clipper, and Seldon Core are used to deploy and manage machine learning models in production.
+3.  *Model Deployment and Management Tools*: Tools such as TensorFlow Serving, AWS Sagemaker, and Seldon Core are used to deploy and manage machine learning models in production.
 4.  *Data Versioning and Management Tools*: Tools such as DVC (Data Version Control) and Pachyderm are used to version and manage the data used for training and evaluating models.    
 5.  *Automated Model Evaluation and Deployment Tools*: Tools such as AlgoTrader and AlgoHub are used to automate the evaluation of machine learning models and deploy the best-performing models to production.
 
 It's important to note that the specific tools used in MLOps and DevOps may vary depending on the organization's needs. Some of the tools are used in both but applied differently in each, e.g. container orchestration tools like Kubernetes. The above lists do also not claim to be complete and there are of course multiple more tools.
 
 ### MLOps Lifecycle
 
-Incorporating the tools introduced by DevOps and MLOps can extend the machine learning workflow outlined in the previous section, resulting in a complete MLOps lifecycle that covers each stage of the machine learning process while integrating automation practices.
+Incorporating the tools introduced by DevOps and MLOps can extend the machine learning workflow outlined in the [previous section](#machine-learning-workflow), resulting in a complete MLOps lifecycle that covers each stage of the machine learning process while integrating automation practices.
 
 ![](images/01-Introduction/mlops-lifecycle.svg){ width=100% }