diff --git a/.gitbook/assets/01_instructions.png b/.gitbook/assets/01_instructions.png new file mode 100644 index 00000000..37f23246 Binary files /dev/null and b/.gitbook/assets/01_instructions.png differ diff --git a/.gitbook/assets/02_instructions.png b/.gitbook/assets/02_instructions.png new file mode 100644 index 00000000..c4154fc8 Binary files /dev/null and b/.gitbook/assets/02_instructions.png differ diff --git a/.gitbook/assets/03_instructions.png b/.gitbook/assets/03_instructions.png new file mode 100644 index 00000000..6335d322 Binary files /dev/null and b/.gitbook/assets/03_instructions.png differ diff --git a/.gitbook/assets/04_instructions.png b/.gitbook/assets/04_instructions.png new file mode 100644 index 00000000..dbd76f3f Binary files /dev/null and b/.gitbook/assets/04_instructions.png differ diff --git a/.gitbook/assets/05_instructions.png b/.gitbook/assets/05_instructions.png new file mode 100644 index 00000000..0ba36d76 Binary files /dev/null and b/.gitbook/assets/05_instructions.png differ diff --git a/.gitbook/assets/06_instructions.png b/.gitbook/assets/06_instructions.png new file mode 100644 index 00000000..8ba45cf4 Binary files /dev/null and b/.gitbook/assets/06_instructions.png differ diff --git a/.gitbook/assets/07_instructions.png b/.gitbook/assets/07_instructions.png new file mode 100644 index 00000000..345bf572 Binary files /dev/null and b/.gitbook/assets/07_instructions.png differ diff --git a/.gitbook/assets/08_instructions.png b/.gitbook/assets/08_instructions.png new file mode 100644 index 00000000..c5b61aa5 Binary files /dev/null and b/.gitbook/assets/08_instructions.png differ diff --git a/.gitbook/assets/09_instructions.png b/.gitbook/assets/09_instructions.png new file mode 100644 index 00000000..0c47634e Binary files /dev/null and b/.gitbook/assets/09_instructions.png differ diff --git a/.gitbook/assets/10_instructions.png b/.gitbook/assets/10_instructions.png new file mode 100644 index 00000000..77a9597d Binary files /dev/null and b/.gitbook/assets/10_instructions.png differ diff --git a/.gitbook/assets/11_instructions.png b/.gitbook/assets/11_instructions.png new file mode 100644 index 00000000..d54e3e57 Binary files /dev/null and b/.gitbook/assets/11_instructions.png differ diff --git a/README.md b/README.md index 63127235..f3f27ec2 100644 --- a/README.md +++ b/README.md @@ -6,42 +6,42 @@ description: Build and manage real-life data science projects with ease. ## Introduction -* [Why Metaflow](introduction/why-metaflow.md) -* [What is Metaflow](introduction/what-is-metaflow.md) -* [Release Notes](introduction/release-notes.md) -* [Roadmap](introduction/roadmap.md) -* [Contributing to Metaflow](introduction/contributing-to-metaflow.md) -* [Get in Touch](introduction/getting-in-touch.md) +- [Why Metaflow](introduction/why-metaflow.md) +- [What is Metaflow](introduction/what-is-metaflow.md) +- [Release Notes](introduction/release-notes.md) +- [Roadmap](introduction/roadmap.md) +- [Contributing to Metaflow](introduction/contributing-to-metaflow.md) +- [Get in Touch](introduction/getting-in-touch.md) ## Getting Started -* [Installing Metaflow](getting-started/install.md) -* [Tutorials](getting-started/tutorials/) +- [Installing Metaflow](getting-started/install.md) +- [Tutorials](getting-started/tutorials/) ## Metaflow on AWS -* [Metaflow on AWS](metaflow-on-aws/metaflow-on-aws.md) -* [Metaflow Sandbox](metaflow-on-aws/metaflow-sandbox.md) -* [Deploying to AWS](metaflow-on-aws/deploy-to-aws.md) +- [Metaflow on AWS](metaflow-on-aws/metaflow-on-aws.md) +- [Metaflow Sandbox](metaflow-on-aws/metaflow-sandbox.md) +- [Deploying to AWS](metaflow-on-aws/deploy-to-aws.md) ## Developing with Metaflow -* [Basics of Metaflow](metaflow/basics.md) -* [Inspecting Flows and Results](metaflow/client.md) -* [Debugging with Metaflow](metaflow/debugging.md) -* [Scaling Out and Up](metaflow/scaling.md) -* [Loading and Storing Data](metaflow/data.md) -* [Managing External Libraries](metaflow/dependencies.md) -* [Dealing with Failures](metaflow/failures.md) -* [Organizing Results](metaflow/tagging.md) +- [Basics of Metaflow](metaflow/basics.md) +- [Inspecting Flows and Results](metaflow/client.md) +- [Debugging with Metaflow](metaflow/debugging.md) +- [Monitoring Flows](metaflow/monitoring.md) +- [Scaling Out and Up](metaflow/scaling.md) +- [Loading and Storing Data](metaflow/data.md) +- [Managing External Libraries](metaflow/dependencies.md) +- [Dealing with Failures](metaflow/failures.md) +- [Organizing Results](metaflow/tagging.md) -## Going to Production with Metaflow +## Going to Production with Metaflow -* [Scheduling Metaflow Flows](going-to-production-with-metaflow/scheduling-metaflow-flows.md) -* [Coordinating Larger Metaflow Projects](going-to-production-with-metaflow/coordinating-larger-metaflow-projects.md) +- [Scheduling Metaflow Flows](going-to-production-with-metaflow/scheduling-metaflow-flows.md) +- [Coordinating Larger Metaflow Projects](going-to-production-with-metaflow/coordinating-larger-metaflow-projects.md) ## Internals of Metaflow -* [Technical Overview](internals-of-metaflow/technical-overview.md) -* [Testing Philosophy](internals-of-metaflow/testing-philosophy.md) - +- [Technical Overview](internals-of-metaflow/technical-overview.md) +- [Testing Philosophy](internals-of-metaflow/testing-philosophy.md) diff --git a/SUMMARY.md b/SUMMARY.md index d36d39ce..bf1fb07c 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,56 +1,56 @@ # Table of contents -* [Welcome to Metaflow for Python](README.md) +- [Welcome to Metaflow for Python](README.md) ## Introduction -* [Why Metaflow](introduction/why-metaflow.md) -* [What is Metaflow](introduction/what-is-metaflow.md) -* [Release Notes](introduction/release-notes.md) -* [Roadmap](introduction/roadmap.md) -* [Contributing to Metaflow](introduction/contributing-to-metaflow.md) -* [Get in Touch](introduction/getting-in-touch.md) +- [Why Metaflow](introduction/why-metaflow.md) +- [What is Metaflow](introduction/what-is-metaflow.md) +- [Release Notes](introduction/release-notes.md) +- [Roadmap](introduction/roadmap.md) +- [Contributing to Metaflow](introduction/contributing-to-metaflow.md) +- [Get in Touch](introduction/getting-in-touch.md) ## Getting Started -* [Installing Metaflow](getting-started/install.md) -* [Tutorials](getting-started/tutorials/README.md) - * [Season 1: The Local Experience](getting-started/tutorials/season-1-the-local-experience/README.md) - * [Episode 0: Hello World](getting-started/tutorials/season-1-the-local-experience/episode00.md) - * [Episode 1: Playlist](getting-started/tutorials/season-1-the-local-experience/episode01.md) - * [Episode 2: Statistics](getting-started/tutorials/season-1-the-local-experience/episode02.md) - * [Episode 3: Playlist Redux](getting-started/tutorials/season-1-the-local-experience/episode03.md) - * [Episode 4: Playlist Plus](getting-started/tutorials/season-1-the-local-experience/episode04.md) - * [Season 2: Scaling Out and Up](getting-started/tutorials/season-2-scaling-out-and-up/README.md) - * [Episode 5: Hello AWS](getting-started/tutorials/season-2-scaling-out-and-up/episode05.md) - * [Episode 6: Statistics Redux](getting-started/tutorials/season-2-scaling-out-and-up/episode06.md) - * [Episode 7: Worldview](getting-started/tutorials/season-2-scaling-out-and-up/episode07.md) - * [Episode 8: Autopilot](getting-started/tutorials/season-2-scaling-out-and-up/episode-8-autopilot.md) +- [Installing Metaflow](getting-started/install.md) +- [Tutorials](getting-started/tutorials/README.md) + - [Season 1: The Local Experience](getting-started/tutorials/season-1-the-local-experience/README.md) + - [Episode 0: Hello World](getting-started/tutorials/season-1-the-local-experience/episode00.md) + - [Episode 1: Playlist](getting-started/tutorials/season-1-the-local-experience/episode01.md) + - [Episode 2: Statistics](getting-started/tutorials/season-1-the-local-experience/episode02.md) + - [Episode 3: Playlist Redux](getting-started/tutorials/season-1-the-local-experience/episode03.md) + - [Episode 4: Playlist Plus](getting-started/tutorials/season-1-the-local-experience/episode04.md) + - [Season 2: Scaling Out and Up](getting-started/tutorials/season-2-scaling-out-and-up/README.md) + - [Episode 5: Hello AWS](getting-started/tutorials/season-2-scaling-out-and-up/episode05.md) + - [Episode 6: Statistics Redux](getting-started/tutorials/season-2-scaling-out-and-up/episode06.md) + - [Episode 7: Worldview](getting-started/tutorials/season-2-scaling-out-and-up/episode07.md) + - [Episode 8: Autopilot](getting-started/tutorials/season-2-scaling-out-and-up/episode-8-autopilot.md) ## Metaflow on AWS -* [Metaflow on AWS](metaflow-on-aws/metaflow-on-aws.md) -* [Metaflow Sandbox](metaflow-on-aws/metaflow-sandbox.md) -* [Deploying to AWS](metaflow-on-aws/deploy-to-aws.md) +- [Metaflow on AWS](metaflow-on-aws/metaflow-on-aws.md) +- [Metaflow Sandbox](metaflow-on-aws/metaflow-sandbox.md) +- [Deploying to AWS](metaflow-on-aws/deploy-to-aws.md) ## Developing with Metaflow -* [Basics of Metaflow](metaflow/basics.md) -* [Inspecting Flows and Results](metaflow/client.md) -* [Debugging with Metaflow](metaflow/debugging.md) -* [Scaling Out and Up](metaflow/scaling.md) -* [Loading and Storing Data](metaflow/data.md) -* [Managing External Libraries](metaflow/dependencies.md) -* [Dealing with Failures](metaflow/failures.md) -* [Organizing Results](metaflow/tagging.md) +- [Basics of Metaflow](metaflow/basics.md) +- [Inspecting Flows and Results](metaflow/client.md) +- [Monitoring Flows](metaflow/monitoring.md) +- [Debugging with Metaflow](metaflow/debugging.md) +- [Scaling Out and Up](metaflow/scaling.md) +- [Loading and Storing Data](metaflow/data.md) +- [Managing External Libraries](metaflow/dependencies.md) +- [Dealing with Failures](metaflow/failures.md) +- [Organizing Results](metaflow/tagging.md) ## Going to Production with Metaflow -* [Scheduling Metaflow Flows](going-to-production-with-metaflow/scheduling-metaflow-flows.md) -* [Coordinating Larger Metaflow Projects](going-to-production-with-metaflow/coordinating-larger-metaflow-projects.md) +- [Scheduling Metaflow Flows](going-to-production-with-metaflow/scheduling-metaflow-flows.md) +- [Coordinating Larger Metaflow Projects](going-to-production-with-metaflow/coordinating-larger-metaflow-projects.md) ## Internals of Metaflow -* [Technical Overview](internals-of-metaflow/technical-overview.md) -* [Testing Philosophy](internals-of-metaflow/testing-philosophy.md) - +- [Technical Overview](internals-of-metaflow/technical-overview.md) +- [Testing Philosophy](internals-of-metaflow/testing-philosophy.md) diff --git a/metaflow/monitoring.md b/metaflow/monitoring.md new file mode 100644 index 00000000..5e56c3b8 --- /dev/null +++ b/metaflow/monitoring.md @@ -0,0 +1,140 @@ +# Monitoring flows with Metaflow UI + +In this section we go through all of the views within Metaflow UI and give you short descriptions what everything is supposed to do. + +## Home + +1. Our **header navigation** provides you shortcuts to quickly navigate to + spesific parts of the site. +2. **Quick links** provides useful features for selecting your timezone and + links to internal and external resources. +3. **Filters** -section provides all the required tools for the user to filter + out their **Runs** -list. +4. **Runs** -table is a realtime overview of the running Flows within your + service. + +> ![Home](../.gitbook/assets/01_instructions.png "Home") + +## Header + +1. Our site search provides the user with **breadcrumb** patterns to quickly + navigate to specific part of the site. +2. Within **Quick links** the user can change the sites timezone. Navigate + to Notifications or external resources. Also the Application and Service + versions are presented here. + +> ![Header](../.gitbook/assets/02_instructions.png "Header") + +## Filters + +1. **Time frame** -filter always has **Last 30 days** set as default to ensure + the best possible user experience. +2. With the **Time frame** -filter you can use quick presets or setup a + custom timeframe. +3. All of the other filter fields have an **autocomplete** function to make the + filtering as easy as possible. Also we display the active filters as **tags** + beneath the filter inputs. +4. In a scenario when a specific part of the service might be temporarily + down the inputs display errors for the user. + +> ![Filters](../.gitbook/assets/03_instructions.png "Filters") + +## "Runs" -table + +1. **Runs** -table is updated in realtime to show the latest status of flows + being run in your service. The list can be ordered by clicking on the + column headers. +2. Each row has **toggle switch** on the right side of the table that appears + when you hover over a row. The toggle opens up a section that + contains the Flows **Parameters** and **Timeline**. +3. You can quickly glance through the flows Parameters and check the + flows current status from the Timeline + +> !["Runs" -table](../.gitbook/assets/04_instructions.png "'Runs' -table") + +## "Runs" -table grouping + +1. You can change the grouping of the **Runs** -table from the **Group by** + -select. Currently you can group by flow or user. +2. When you enable grouping we show a **maximum of six** of the latest + runs in a group. The table functions the same way as described + before. +3. Whenever there’s more than six items in a group we provide a **Show + all runs** -link that will filter the **Runs** -table to only show all of the + flows within the group. +4. When you scroll through the **Runs** -table we’ll bring forth a **Scroll to + top** -button to easily get back to top of the table. + +> !["Runs" -table grouping](../.gitbook/assets/05_instructions.png "'Runs' -table grouping") + +## Notifications + +1. Notifications is a simple view displaying the history of past + **announcements** related to you service. +2. Each notification consists of a simple message, publication date and a + possible external resource link. +3. An example notification that can popup at any given time anywhere on + the page, to give you valuable information about upcoming + maintenances or software updates. + +> ![Notifications](../.gitbook/assets/06_instructions.png "Notifications") + +## Run details - Timeline + +1. **Run details** -header contains all of the vital information about you + current run. +2. Underneath details you’ll find runs **parameters** and **tags**. +3. The tab navigation allows you to navigate trough runs DAG, Timeline + and Task views. +4. **Step collapse** -button allows you to quickly collapse or close the + steps within a run. The **Artifact** search input enables you to filter tasks + with key:value pair searches. +5. **Mode** -select provides you quick presets to filter the timeline. +6. **Order by** -select lets you change the order of the timeline +7. **Status** -select allows you to filter the timeline by task status. +8. Timelines **zoom controls**. +9. **Steps and tasks** display a realtime status of how the run is progressing. +10. The **timeline** displays a graphical presentation of the runs progress in + realtime. + +> ![Run details - Timeline](../.gitbook/assets/07_instructions.png "Run details - Timeline") + +## Run details - DAG + +1. DAG -view **fullscreen toggle**. +2. Runs **DAG map** that gives graphical presentation how the different + steps are separated and how they’re progressing within the run. +3. You can also check the **Developer comments** for each step if the + developer has provided any. + +> ![Run details - DAG](../.gitbook/assets/08_instructions.png "Run details - DAG") + +## Run details - Task details 1/3 + +1. Each task can consist of multiple attempts. **Task details** -header + beneath the tab navigation contains all of the vital information related + a to specific task within a run. +2. **Task details** -section contains information related to the tasks + **Metadata**, **Developer comment** and **Download link**. +3. Metadata is automatically generated data related to the step. + +> ![Run details - Task details 1/3](../.gitbook/assets/09_instructions.png "Run details - Task details 1/3") + +## Run details - Task details 2/3 + +1. Task related **Developer comments** can be found here. +2. The tasks **Download link** can be found here. + +> ![Run details - Task details 2/3](../.gitbook/assets/10_instructions.png "Run details - Task details 2/3") + +## Run details - Task details 3/3 + +1. **Std out** -section displays the tasks std out if available and also + provide tools for downloading or copying the outcome to your + clipboard. +2. **Std err** -section displays the tasks std err if available. +3. **Artifacts** -table contains all of the artifacts used within the step. +4. You can copy the Python, R or original raw url for each artifact found + within the **Artifacts** -table. + +> ![Run details - Task details 3/3](../.gitbook/assets/11_instructions.png "Run details - Task details 3/3")