pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Run a pipeline for a campaign"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Instructions:\n",
    "\n",
    "This \"notebook\" allows a user to work through all the steps in a campaign. Just go through the steps one-by-one. To execute a step, click on it, and then click on the \"Run\" button. \n",
    "\n",
    "You can stop at any point, and then return to the notebook at any later time. If you do, first run the first initialization section.\n",
    "\n",
    "All the output and plots will stay in between the sessions. To clean everything up, press \"Cell\" in the menu bar -> \"All Output\" -> \"Clear\"."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Initialization. Run every time.\n",
    "\n",
    "Run it every time the notebook starts."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "%matplotlib inline\n",
    "import os\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import argparse\n",
    "import glob\n",
    "import shutil"
   ],
   "outputs": [],
   "metadata": {
    "scrolled": true
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Input data"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "CAMPAIGN_ID = 7"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Start campaign\n",
    "\n",
    "Create a new campaign and pick images."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/select_new_campaign.sh --num_images_in_campaign 50 --out_version 1 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Detect stamps using the best model"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/detect_stamps.sh --in_version 1 --out_version 2 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Detect pages using the best model"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/detect_stamps.sh --in_version 2 --out_version 3 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Crop stamps to run classification"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/crop_stamps_for_classification.sh --in_version 3 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Run the classification"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/classify_stamps.sh "
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Import the classification results."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/import_classification_results.sh --in_version 3 --out_version 4 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Visualize ML predictions."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/visualize_ml_predictions.sh --in_version 4 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Labeling in Labelme\n",
    "\n",
    "Export to Labelme"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/export_to_labelme.sh --in_version 4 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Upload to the LabelMe server"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/upload_to_labelme_server.sh --name \"initial\" --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Please go to https://vm019.bridges2.psc.edu/ and label data."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Download from the Labelme server"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/download_from_labelme_server.sh --name \"initial\" --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Import labeled data from Labelme"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/import_from_labelme.sh --out_version 5 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Display statistics. It will help to decide if more cleaning or more custom rules are needed."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/display_statitics.sh --in_version 5 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Cleaning in Labelme\n",
    "\n",
    "#### For every round of cleaning, repeat the following steps. "
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Every new time you do the cleaning, increment CLEANING_ITERATION."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "CLEANING_ITERATION = 1"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Export to labelme as \"collage\" tiles for cleaning. May take 10-30 mins."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/export_to_labelme_cleaning.sh --in_version 5 --subversion {CLEANING_ITERATION} --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Upload to the Labelme server"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/upload_to_labelme_server.sh --name cleaning{CLEANING_ITERATION} --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Please go to https://vm019.bridges2.psc.edu/ and clean data."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Download from the Labelme server. **Make sure to first run the cell `CLEANING_ITERATION =` (above).**"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/download_from_labelme_server.sh --name cleaning{CLEANING_ITERATION} --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Import from labelme after cleaning"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/import_from_labelme_cleaning.sh --in_version 5 --out_version 6 --subversion {CLEANING_ITERATION} --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Apply special rules. Rename class and delete class is currently supported"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/custom_cleaning.sh --rename_class \"\\\"{'dog': 'cat', 'sheep': 'panda'}\\\"\"  --in_version 6 --out_version 7 --subversion {CLEANING_ITERATION} --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Display statistics. It will help to decide if more cleaning or more custom rules are needed."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/display_statitics.sh --in_version 7 --subversion {CLEANING_ITERATION} --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finalize cleaned."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/promote_cleaning_iteration.sh --in_version 7 --subversion {CLEANING_ITERATION}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### No more cleaning after this step."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Training ML models\n",
    "\n",
    "Start STAMP detection training. It may take 4-12 hours. Run the next step in parallel to this one."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/start_stamp_detection_training.sh --in_version 7 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Start PAGE detection training. It may take 4-12 hours. Run it in parallel with the prevoius step."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/start_page_detection_training.sh --in_version 7 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Please wait until the training is complete\n",
    "\n",
    "Check if all training has completed."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/check_if_all_jobs_have_completed.sh"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finalize STAMP detection training."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/finalize_stamp_detection_training.sh --in_version 7 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finalize PAGE detection training."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/finalize_page_detection_training.sh --in_version 7 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Evaluate detectors trained on previous campaigns on this campaign"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/evaluate_previous_detectors_on_this.sh --in_version 7 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Crop out stamps to train the classification model"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/crop_stamps_to_train_classifier.sh --in_version 7 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Start the classification training"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/start_stamp_classification_training.sh --in_version 7 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Check if all training has completed."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/check_if_all_jobs_have_completed.sh"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Release the MiikeMineStamps dataset\n",
    "\n",
    "Make a release of MiikeMineStamps dataset. Its README files need to be changed separately manually."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "!pipeline/release_miike_mine_stamp_dataset.sh --in_version 7 --campaign_id {CAMPAIGN_ID}"
   ],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "dd10b49f89a3e8c1303ad5078ae3ab4f538af7825216d89830b005a85f025a3e"
  },
  "kernelspec": {
   "display_name": "shuffler in ikernel",
   "language": "python",
   "name": "shuffler"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}