tutorials/bicycle_demand_monitoring_setup.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Monitoring setup for Bicycle Sharing Demand Prediction"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "QgumzOZ5wgec"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This notebook shows how you can use the Evidently to:\n",
        "* calculate performance and data drift for the model, performed as batch checks\n",
        "* log models quality & data drift using MLflow Tracking\n",
        "* explore the result\n",
        "\n",
        "More examples are available in the github: https://github.com/evidentlyai/evidently/tree/main/examples\n",
        "\n",
        "Evidently docs: https://docs.evidentlyai.com/\n",
        "\n",
        "Join our Discord: https://discord.com/invite/xZjKRaNp8b"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "6eM1BMvDwgee"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "try:\n",
        "    import evidently\n",
        "except:\n",
        "    !pip install evidently==0.5.1"
      ],
      "outputs": [],
      "metadata": {
        "id": "oDEk9bFNwgeh",
        "pycharm": {
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "import datetime\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import requests\n",
        "import zipfile\n",
        "import io\n",
        "import json\n",
        "\n",
        "from sklearn import datasets, ensemble, model_selection\n",
        "from scipy.stats import anderson_ksamp\n",
        "\n",
        "from evidently.metrics import RegressionQualityMetric, RegressionErrorPlot, RegressionErrorDistribution\n",
        "from evidently.metric_preset import DataDriftPreset, RegressionPreset\n",
        "from evidently.pipeline.column_mapping import ColumnMapping\n",
        "from evidently.report import Report"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "rByuPhg7wgei"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "warnings.simplefilter('ignore')"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "HiiUl3p8wgej"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Bicycle Demand Data"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "zw5Tap_Xwgej"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "More information about the dataset can be found in UCI machine learning repository: https://archive.ics.uci.edu/ml/datasets/bike+sharing+dataset\n",
        "\n",
        "Acknowledgement: Fanaee-T, Hadi, and Gama, Joao, 'Event labeling combining ensemble detectors and background knowledge', Progress in Artificial Intelligence (2013): pp. 1-15, Springer Berlin Heidelberg"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "VqGH1Mr6wgej"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "content = requests.get(\"https://archive.ics.uci.edu/static/public/275/bike+sharing+dataset.zip\", verify=False).content\n",
        "with zipfile.ZipFile(io.BytesIO(content)) as arc:\n",
        "    raw_data = pd.read_csv(arc.open(\"hour.csv\"), header=0, sep=',', parse_dates=['dteday'])"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "dKX2YV19wgek"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "raw_data.index = raw_data.apply(lambda row: datetime.datetime.combine(row.dteday.date(), datetime.time(row.hr)),\n",
        "                                axis=1)"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "36Gk-YMhwgek"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "raw_data.head()"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "N6oQxQKNwgek"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model training"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "dhZOCJZ1wgel"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "target = 'cnt'\n",
        "prediction = 'prediction'\n",
        "numerical_features = ['temp', 'atemp', 'hum', 'windspeed', 'mnth', 'hr', 'weekday']\n",
        "categorical_features = ['season', 'holiday', 'workingday', ]#'weathersit']"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "_i8edS6Ewgem"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "reference = raw_data.loc['2011-01-01 00:00:00':'2011-01-28 23:00:00']\n",
        "current = raw_data.loc['2011-01-29 00:00:00':'2011-02-28 23:00:00']"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "vTMu0TS_wgem"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "X_train, X_test, y_train, y_test = model_selection.train_test_split(\n",
        "    reference[numerical_features + categorical_features],\n",
        "    reference[target],\n",
        "    test_size=0.3\n",
        ")"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "dqiRpf2Swgem"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "regressor = ensemble.RandomForestRegressor(random_state = 0, n_estimators = 50)"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "PWJv-0UKwgen"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "regressor.fit(X_train, y_train)"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "UOWSxRh1wgen"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "preds_train = regressor.predict(X_train)\n",
        "preds_test = regressor.predict(X_test)"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "TZKZ_biHwgen"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model validation"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "pRV2_SNSwgen"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "X_train['target'] = y_train\n",
        "X_train['prediction'] = preds_train\n",
        "\n",
        "X_test['target'] = y_test\n",
        "X_test['prediction'] = preds_test"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "JcEFJgEswgeo"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "column_mapping = ColumnMapping()\n",
        "\n",
        "column_mapping.target = 'target'\n",
        "column_mapping.prediction = 'prediction'\n",
        "column_mapping.numerical_features = numerical_features\n",
        "column_mapping.categorical_features = categorical_features"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "wEhoX-Xcwgeo"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "regression_performance_report = Report(metrics=[\n",
        "    RegressionPreset(),\n",
        "])\n",
        "\n",
        "regression_performance_report.run(reference_data=X_train.sort_index(), current_data=X_test.sort_index(),\n",
        "                                  column_mapping=column_mapping)\n",
        "regression_performance_report"
      ],
      "outputs": [],
      "metadata": {
        "id": "iQk44skSwgeo",
        "pycharm": {
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Production model training"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "r4QPdp17wgeo"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "regressor.fit(reference[numerical_features + categorical_features], reference[target])"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "9u3zyR-bwgep"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "column_mapping = ColumnMapping()\n",
        "\n",
        "column_mapping.target = target\n",
        "column_mapping.prediction = prediction\n",
        "column_mapping.numerical_features = numerical_features\n",
        "column_mapping.categorical_features = categorical_features"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "qH5FK4x8wgep"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "ref_prediction = regressor.predict(reference[numerical_features + categorical_features])\n",
        "reference['prediction'] = ref_prediction"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "vTCmG4Ntwgeq"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "regression_performance_report = Report(metrics=[\n",
        "    RegressionPreset(),\n",
        "])\n",
        "\n",
        "regression_performance_report.run(reference_data=None, current_data=reference,\n",
        "                                  column_mapping=column_mapping)\n",
        "regression_performance_report"
      ],
      "outputs": [],
      "metadata": {
        "id": "uV1kyj3Owgeq",
        "pycharm": {
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Some time has passed - how is the model working?"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "2rJyfTJ3wger"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "current_prediction = regressor.predict(current[numerical_features + categorical_features])\n",
        "current['prediction'] = current_prediction"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "TqljYICLwger"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Week 1"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "7eBBRAOTwger"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "regression_performance_report.run(reference_data=reference, current_data=current.loc['2011-01-29 00:00:00':'2011-02-07 23:00:00'],\n",
        "                                  column_mapping=column_mapping)\n",
        "regression_performance_report"
      ],
      "outputs": [],
      "metadata": {
        "id": "LvuPle_Nwges",
        "pycharm": {
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Week 2"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "SbmG0hvSwges"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "regression_performance_report = Report(metrics=[\n",
        "    RegressionQualityMetric(),\n",
        "    RegressionErrorPlot(),\n",
        "    RegressionErrorDistribution()\n",
        "])\n",
        "\n",
        "regression_performance_report.run(reference_data=reference, current_data=current.loc['2011-02-07 00:00:00':'2011-02-14 23:00:00'],\n",
        "                                            column_mapping=column_mapping)\n",
        "regression_performance_report.show()"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "scrolled": false,
        "id": "d6tBIZDUwget"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Week 3"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "Zj3AHeU-wget"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "regression_performance_report = Report(metrics=[\n",
        "    RegressionQualityMetric(),\n",
        "    RegressionErrorPlot(),\n",
        "    RegressionErrorDistribution()\n",
        "])\n",
        "\n",
        "regression_performance_report.run(reference_data=reference, current_data=current.loc['2011-02-15 00:00:00':'2011-02-21 23:00:00'],\n",
        "                                            column_mapping=column_mapping)\n",
        "regression_performance_report.show()"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "LF6ZFyx_wgeu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Why the quality has dropped?"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "sFeB72oIwgev"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "column_mapping_drift = ColumnMapping()\n",
        "\n",
        "column_mapping_drift.target = target\n",
        "column_mapping_drift.prediction = prediction\n",
        "column_mapping_drift.numerical_features = numerical_features\n",
        "column_mapping_drift.categorical_features = []"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "ubmWrELOwgev"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "data_drift_report = Report(metrics=[\n",
        "    DataDriftPreset(),\n",
        "])\n",
        "\n",
        "data_drift_report.run(\n",
        "    reference_data=reference,\n",
        "    current_data=current.loc['2011-02-14 00:00:00':'2011-02-21 23:00:00'],\n",
        "    column_mapping=column_mapping_drift,\n",
        ")\n",
        "data_drift_report"
      ],
      "outputs": [],
      "metadata": {
        "id": "Uz1ajc22wgew",
        "pycharm": {
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Let's customize the report!"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "-4S20imIwgew"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "from evidently.core import ColumnType\n",
        "from evidently.calculations.stattests import StatTest, register_stattest\n",
        "\n",
        "def _anderson_stat_test(reference_data: pd.Series, current_data: pd.Series, feature_type: ColumnType, threshold: float):\n",
        "    p_value = anderson_ksamp([reference_data, current_data])[2]\n",
        "    return p_value, p_value < threshold\n",
        "\n",
        "anderson_stat_test = StatTest(\n",
        "    name=\"anderson\",\n",
        "    display_name=\"Anderson test (p_value)\",\n",
        "    allowed_feature_types=[ColumnType.Numerical]\n",
        ")\n",
        "\n",
        "register_stattest(anderson_stat_test, default_impl=_anderson_stat_test)\n",
        "# options = DataDriftOptions(feature_stattest_func=anderson_stat_test, all_features_threshold=0.9, nbinsx=20)"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "zJWuGbzdwgew"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "the_report = Report(metrics=[\n",
        "    RegressionQualityMetric(),\n",
        "    RegressionErrorPlot(),\n",
        "    RegressionErrorDistribution(),\n",
        "    DataDriftPreset(stattest=anderson_stat_test, stattest_threshold=0.9),\n",
        "])\n",
        "\n",
        "\n",
        "the_report.run(\n",
        "    reference_data=reference,\n",
        "    current_data=current.loc['2011-02-14 00:00:00':'2011-02-21 23:00:00'],\n",
        "    column_mapping=column_mapping_drift\n",
        ")\n",
        "the_report"
      ],
      "outputs": [],
      "metadata": {
        "id": "EDqb_S9Xwgex",
        "pycharm": {
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Jupyter vs Automated Run?"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "q5lW24Xzwgex"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To run this part of the tutorial you need to install MLflow (if not installed yet)\n",
        "\n",
        "You can install MLflow with the following command: `pip install mlflow` or install MLflow with scikit-learn via `pip install mlflow[extras]`\n",
        "\n",
        "More details:https://mlflow.org/docs/latest/tutorials-and-examples/tutorial.html#id5"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "p-80D3Bbwgex"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "try:\n",
        "    import mlflow\n",
        "except:\n",
        "    !pip install mlflow"
      ],
      "outputs": [],
      "metadata": {
        "id": "bUoXVw40AKWj",
        "pycharm": {
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "import mlflow\n",
        "#import mlflow.sklearn\n",
        "from mlflow.tracking import MlflowClient"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "I_IyYlM0wgey"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "experiment_batches = [\n",
        "    ('2011-01-29 00:00:00','2011-02-07 23:00:00'),\n",
        "    ('2011-02-07 00:00:00','2011-02-14 23:00:00'),\n",
        "    ('2011-02-15 00:00:00','2011-02-21 23:00:00'),\n",
        "]"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "L6PKtAGEwgey"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "the_report = Report(metrics=[\n",
        "    RegressionQualityMetric(),\n",
        "    RegressionErrorPlot(),\n",
        "    RegressionErrorDistribution(),\n",
        "    DataDriftPreset(stattest=anderson_stat_test, stattest_threshold=0.9),\n",
        "])\n",
        "\n",
        "the_report.run(\n",
        "    reference_data=reference,\n",
        "    current_data=current.loc[experiment_batches[0][0]:experiment_batches[0][1]],\n",
        "    column_mapping=column_mapping_drift\n",
        ")"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "nHegk6fAwgey"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "logged_json = json.loads(the_report.json())"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "scrolled": false,
        "id": "qMD5Ek7-wgey"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "logged_json"
      ],
      "outputs": [],
      "metadata": {
        "id": "xwOcfczR-b6d",
        "pycharm": {
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "[x['metric'] for x in logged_json['metrics']]"
      ],
      "outputs": [],
      "metadata": {
        "id": "cyVps9-Q-ZoB",
        "pycharm": {
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "logged_json['metrics'][0]['result']['current']['mean_error']"
      ],
      "outputs": [],
      "metadata": {
        "id": "yc0HhabLxYVj",
        "pycharm": {
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "logged_json['metrics'][3]['result']['drift_share']"
      ],
      "outputs": [],
      "metadata": {
        "id": "SolMtKPjyHw5",
        "pycharm": {
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "#log into MLflow\n",
        "client = MlflowClient()\n",
        "\n",
        "#set experiment\n",
        "mlflow.set_experiment('Model Quality Evaluation')\n",
        "\n",
        "#start new run\n",
        "for date in experiment_batches:\n",
        "    with mlflow.start_run() as run: #inside brackets run_name='test'\n",
        "\n",
        "        # Log parameters\n",
        "        mlflow.log_param(\"begin\", date[0])\n",
        "        mlflow.log_param(\"end\", date[1])\n",
        "\n",
        "        # Get metrics\n",
        "        the_report = Report(metrics=[\n",
        "            RegressionQualityMetric(),\n",
        "            RegressionErrorPlot(),\n",
        "            RegressionErrorDistribution(),\n",
        "            DataDriftPreset(stattest=anderson_stat_test, stattest_threshold=0.9),\n",
        "        ])\n",
        "        the_report.run(\n",
        "            reference_data=reference,\n",
        "            current_data=current.loc[date[0]:date[1]],\n",
        "            column_mapping=column_mapping_drift)\n",
        "        logged_json = json.loads(the_report.json())\n",
        "\n",
        "        me = logged_json['metrics'][0]['result']['current']['mean_error']\n",
        "        mae = logged_json['metrics'][0]['result']['current'][\"mean_abs_error\"]\n",
        "        drift_share = logged_json['metrics'][3]['result']['drift_share']\n",
        "\n",
        "        # Log metrics\n",
        "        mlflow.log_metric('me', round(me, 3))\n",
        "        mlflow.log_metric('mae', round(mae, 3))\n",
        "        mlflow.log_metric('drift_share', round(drift_share, 3))\n",
        "\n",
        "        print(run.info)"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "bTHU8eAqwgez"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "source": [
        "#run MLflow UI (NOT recommended! It will be more convinient to run it directly from the TERMINAL)\n",
        "#!mlflow ui"
      ],
      "outputs": [],
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "IUYxYN5nwgez"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Support Evidently\n",
        "Enjoyed the tutorial? Star Evidently on GitHub to contribute back! This helps us continue creating free open-source tools for the community. https://github.com/evidentlyai/evidently"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "zCEGUaYRnP-W"
      }
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": false,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": false
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}