dataset_annotation/segmentation_dataset/segmentation_standard.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Segmentation datasets annotation standard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The standard describes:\n",
    "* components of annotation specifications;\n",
    "* recommendations on content of the componenst and their design;\n",
    "* methods for validating annotation;\n",
    "* annotation tools and services.\n",
    "\n",
    "The actions recommended by the standard are aimed to:\n",
    "* increase reproducibility of dataset generation;\n",
    "* improve the quality of labels.\n",
    "\n",
    "**The description should be sufficient enough to allow a new analyst to:**\n",
    "* **write a parser for data and annotations;**\n",
    "* **generate new consistent dataset;**\n",
    "* **annotatate new dataset in the same way.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of contents\n",
    "\n",
    "* [Segmentation](#Segmentation)\n",
    "    * [Segmentation task](#Segmentation-task)\n",
    "    * [Applications](#Applications)\n",
    "    * [Types of segmentation](#Types-of-segmantation)\n",
    "* [Dataset description](#Dataset-description)\n",
    "* [Annotation description](#Annotation-description)\n",
    "    * [Annotation dictionary](#Annotation-dictionary)\n",
    "    * [Annotation guidelines](#Annotation-guidelines)\n",
    "    * [Instructions for annotators](#Instructions-for-annotators-(text-+-video))\n",
    "    * [Examples of annotation](#Learning-examples---good-and-bad-annotations)\n",
    "    * [Annotation format](#Annotation-format)\n",
    "    * [Number of annotators](#Number-of-annotators)\n",
    "    * [Annotation validation](#Annotation-validation)\n",
    "* [Annotation tools](#Annotation-tools)\n",
    "    * [CVAT](#CVAT)\n",
    "    * [LabelMe](#LabelMe)\n",
    "    * [LabelBox](#LabelBox)\n",
    "    * [Supervise.ly](#Supervise.ly)\n",
    "    * [Yandex.Toloka](#Yandex.Toloka)\n",
    "    * [DBrain](#DBrain)\n",
    "    * [Playment.io](#Playment.io)\n",
    "* [Annotation description example](#Annotation-description-example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Segmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Segmentation task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Segmentation is the process of splitting an image into groups of pixels, according to the proximity of their characteristics. Segmentation also can be considered as classification of each pixel in the image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Segmentation is often applied to a wide range of tasks, because it allows to retrieve a very large amount of information about objects in the image.\n",
    "\n",
    "Examples of common areas and tasks for segmentation:\n",
    "\n",
    "* Medical image analysis\n",
    "    * detection and localization of tumors\n",
    "    * determination of tissue volumes\n",
    "* Analysis of satellite images\n",
    "    * building a road graph\n",
    "    * car detection\n",
    "    * detection of illegal buildings\n",
    "* Self-driving cars\n",
    "    * detection of pedestrians in sight\n",
    "    * detection of other road users\n",
    "    * identification of surrounding objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Types of segmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Semantic segmentation**\n",
    "\n",
    "In semantic segmentation, pixels are combined into groups according to a semantic attribute. For example, pedestrians, cars, chairs, dogs are four different groups of pixels; one group can include to many objects without differentiation across then, as in the example below:\n",
    "\n",
    "<img src=\"./images/Ex_SS.jpg\" style=\"width: 700px;\"/>\n",
    "\n",
    "* **Instance segmentation**\n",
    "\n",
    "Instance segmentation is a bit more difficult task. During instance segmentation, individual objects in the image are identified and classified. Thus, even objects of the same type, such as cars or chairs, will be detected separately, as in an example:\n",
    "\n",
    "<img src=\"./images/Ex_IS.jpg\" style=\"width: 700px;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main purpose of dataset description is to ensure reproducibility of data collection and preparation. This may be required to update, enlarge data or re-collect data when it is lost.\n",
    "\n",
    "The dataset description should be produced in the process of data collection/preparation or immediately after, regardless of the labeling process. Later this description can be updated with labeling procedures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data source\n",
    "Data source description should include:\n",
    "* Data owner - the legal entity or division that owns the data; an employee who can provide access to the data. You should also specify the method of communication with the data owner.\n",
    "* Data storage - if possible, you should indicate where the data is stored: in which databases / tables, in which storages, etc.\n",
    "* The labeling process - the documents described below.\n",
    "\n",
    "#### Obtaining the data\n",
    "It is necessary to describe the procedure for obtaining the data, which may include: filing a request to data owner; coordination with the security department; anonymization process, etc.\n",
    "\n",
    "#### Format and dataset size\n",
    "It is necessary to describe the formats of files the dataset and its size. It should describe in which files the data is stored and whether any transformations were made; it should include description of the metadata. Dataset annotation should be described in the same way. It is advides to specify the distribution of classes in the data.\n",
    "\n",
    "E.g.:\n",
    "\n",
    "`\n",
    "Data is stored in .jpg files. It represents each 48th frame of the original_video.avi, starting from 1: 23: 45.678. This procedure resulted in 12345 images. Dataset is labeled for the multi-class segmentation task: class \"background\" is represented by 75% of the pixels, class \"person\" by 21%, class \"helmet\" by 4%. The annotation for each image is stored in xml format.\n",
    "`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Annotation description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Annotation dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main objective of the dictionary is to clarify characteristics of the objects that should be used to assign the object to a certain class\n",
    "For example, when labeling aerial images, lakes and rivers can classified as `Water`, while ponds located on the territory of cities can be can be classified as `Populated areas`.\n",
    "\n",
    "The dictionary allows you to distinguish several semantically close objects into different classes of combine them in one class. The dictionary is the first and most high-level part of the lableing documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|<p align=\"left\">Class|<p align=\"left\">Description|\n",
    "|---|---|\n",
    "|<p align=\"left\">Dense forest|<p align=\"left\">Forest which does not allow to see the ground|\n",
    "|<p align=\"left\">Rare forest|<p align=\"left\">Forest allows to see the ground|\n",
    "|<p align=\"left\">Ground|<p align=\"left\">Flat surface without large vegetation, not a swamp|\n",
    "|<p align=\"left\">Water|<p align=\"left\">Lakes, rivers|\n",
    "|<p align=\"left\">Swamp|<p align=\"left\">Swamps, marshes, possibly surroundings of the lakes|\n",
    "|<p align=\"left\">Rocks|<p align=\"left\">Territory with stones scattered in a significant amount, difficult to pass on transport|\n",
    "|<p align=\"left\">Road|<p align=\"left\">Any visible roads: asphalt, gravel, etc .; parkings along the roads|\n",
    "|<p align=\"left\">Populated areas|<p align=\"left\">Territories with inhabitant buildings and related infrastructure|\n",
    "|<p align=\"left\">Exclusive areas|<p align=\"left\">Pipelines, power lines, drilling and production sites|\n",
    "|<p align=\"left\">Other|<p align=\"left\">All other objects|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Annotation guidelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Should be prepared beforehand and enhanced during the labeling process.\n",
    "\n",
    "Must answer frequently asked questions that annotators may have, solve uncertainties and make recommendations on specific cases:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|<p align=\"left\">Question|<p align=\"left\">Answer|\n",
    "|---|---|\n",
    "|<p align=\"left\">How to deal with images with poor quality?|<p align=\"left\">If it is impossible to identify the classes described in the dictionary, you should not label it and mark the photo as incorrect.|\n",
    "|<p align=\"left\">How to deal with objects covered with clouds?|<p align=\"left\">If an object is covered by a cloud in a way that the boundaries of the object are indistinguishable, or if the cloud covers more than 15% of the object you should draw objects' boundaries along the cloud boundary. If the cloud does not intersect with the boundaries of the object and occupies less than 15% of the object, it can be considered part of the object.|\n",
    "|<p align=\"left\">How to deal with concatenated images, collages?|<p align=\"left\">If the photos are concatenated so that the objects match (e.g. the coastline and the roads are continuous), then the labeling is carried out as usual. If not, the photo should be marked as incorrect.|\n",
    "|<p align=\"left\">How precise the labels should be?|<p align=\"left\">The labeling should be carried out with accuracy of 5 pixels. This means that all pixels inside the border should belong only to the class object. Pixels more than 5 pixels away from the border should belong to a background or another class. The border and the interval of 5 pisels from it to the outside can belong to both the class object and other objects / background.|\n",
    "|<p align=\"left\">How to label blurry / shuffled pixels?|<p align=\"left\">If pixel class is not clear due to the blurring or transparency of objects, the pixel should be assigned to the class that fits closer to the color of the pixel.|\n",
    "|<p align=\"left\">What to do with intersecting / overlapping objects?|<p align=\"left\">If objects overlap (for example, a road or railway bridge that goes across a river), then priority classes should be marked first, if any. Then, in order from the top / closest object (in the example - the bridge) to the lower / farther.\n",
    "|<p align=\"left\">Shoul small structures be labeled?|<p align=\"left\">If possible, you should try to mark small structures (3-5 pixels). If the small structure belongs to the priority class object, then such a structure must be labeled.|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instructions for annotators (text + video)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should prepare instructions for annotators, both text and video.\n",
    "\n",
    "The text instruction should describe the process using the annotation software, including:\n",
    "* data download (if required);\n",
    "* tools of the annotation software, which should be used by the annotator (for example, \"Brush\", \"Fill\", \"Polygon\" etc.);\n",
    "* annotation process;\n",
    "* annotation saving (if required);\n",
    "\n",
    "In video instructions, you need to show the process described above with comments. Can be done as a screencast."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning examples - good and bad annotations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When designing an annotation task you are highly recommended to create a set of good training examples. It should contain at least a few dozen very carefully labeled images. With this examples, annotators can get familiar with annotation classes and understand details described in annotation guidelines.\n",
    "\n",
    "Some of these examples can be given to annotators as tasks and used as ground truth to evaluate annotators performance.\n",
    "\n",
    "In addition to good examples, you should create a set of bad examples to demonstrate typical errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Annotation format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most popular formats to store annotations:\n",
    "* json\n",
    "* xml\n",
    "* png / jpg / jpeg\n",
    "* csv / tsv\n",
    "\n",
    "Different tools can support one or more formats. The internal structure of the json, xml, csv files differs from tool to tool and for each new tool you need to create a new script to load annotation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Number of annotators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When training models, the quality of input data is very important - garbage in, garbage out. Therefore, you need a large amount of quality data.\n",
    "\n",
    "To select the number of annotators *L* required to label one image:\n",
    "1. Use teaching examples to assess the quality of annotators.\n",
    "2. Select annotators that meet quality requirements and follow the instructions.\n",
    "3. Evaluate how many such annotators should annotate single image to get required quality.\n",
    "\n",
    "The choice of the total number of annotators * K * should be done as follows:\n",
    "1. Estimate time *t* required for one annotator to label a single image.\n",
    "2. Let *T* be the total time allocated for annotation.\n",
    "3. The number of annotators is calculated by the formula:\n",
    "\n",
    "$$ K = \\frac{t * N * L} {T},$$\n",
    "\n",
    "where *N* is the total number of images in dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Annotation validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to use the annotated dataset to train models, you need to validate the annotation.\n",
    "\n",
    "There are several ways to do it:\n",
    "\n",
    "* By intersection - using values such as PPV, DICE, IoU.\n",
    "* By distance - Mean Distance, Hausdorf distance.\n",
    "* Voting methods.\n",
    "* Using annotators scores from learning examples.\n",
    "* Validation by eye.\n",
    "\n",
    "These methods can be applied separately and in combination.\n",
    "\n",
    "For example, consider a two-stage annotation process of LUNA16 dataset performed by four radiologists: in the initial blinded-read phase, each radiologist independently reviewed each CT scan and marked lesions; in the subsequent unblinded-read phase, each radiologist independently reviewed their own marks along with the anonymized marks of the three other radiologists to render a final opinion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Annotation tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The annotation tool is chosen based on the task, the availability of computational and human resources, and data confidentiality.\n",
    "\n",
    "For example, if you have your own server, confidential dataset and a large number of annotators, you can choose CVAT, since it scales well and data does leave your server. If the dataset is open, it is small and the number of annotators is also small, you can choose supervise.ly or LabelBox.\n",
    "\n",
    "The selected tool (and its specific settings, if any), must be included in the annotation description to ensure reproducibility of the process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CVAT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Powerful tool, suitable for large-scale annotation. The server is packed in a docker container that can be set up in an hour. The service has different roles: administrator who creates and edits jobs, annotator, etc.\n",
    "\n",
    "Labels, lines, bboxes and polygons can be used for annotation, which is exported to xml files.\n",
    "\n",
    "Has an extensive [user manual](https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/user_guide.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LabelMe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simple tool, easy to install and run on local machine.\n",
    "Allows you to make markup with polygons; convenient for instance segmentation.\n",
    "The annotation is exported to json, but it can be converted to mask ing png format, original image and additional files — list of classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LabelBox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An online platform; allows to annotate a few thousand images for free each year. You can configure the interface to customize annotation process.\n",
    "\n",
    "It has a convenient pixel-level annotation interface - a magic brush that fills in areas (like Photoshop). Magic brush sometimes reduces the accuracy, but allows you to go beyond the polygons, to make discontinuous, ragged masks; also greatly accelerates the annotation process.\n",
    "\n",
    "The platform allow to use consensus of annotations, but for segmentation it works oddly.\n",
    "\n",
    "It is quite difficult to download annotation - exports json and csv files with a bunch of links to png masks and markup.\n",
    "\n",
    "There are examples of annotation of the original image (in the middle) with a LabelBox magic brush (on the right) and Supervise.ly polygons (on the left)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./images/1_compare.jpg\" style=\"width: 1500px;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Supervise.ly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convenient online tool. Allows you to upload images without any accompanying files. You can have several datasets within one project, and classes between them can be shared.\n",
    "\n",
    "It has polygons and bitmap filling for the segmentation. Allows you to download masks, as well as json files with classes and polygon points.\n",
    "\n",
    "In the examples below, the image was annotated for classes \"Water\", \"Forest\", \"Mountains\" and \"Other.\" The process took about 5-10 minutes.\n",
    "\n",
    "Mask, original image and overlay:\n",
    "<img src=\"./images/2_compare.jpg\" style=\"width: 1200px;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Yandex.Toloka"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Online platform that allows you to perform a number of annotation tasks - labeling, segmentation, bboxes, comparing pictures side by side, walking tasks.\n",
    "\n",
    "It has customizable interfaces; the price for the markup is set by the user - you can speed up the process by increasing the price and attracting more annotators. The quality of annotations is low - out of five annotations of asmall image 2 were performed moderately, one badly and two did not even make sense. A big image had only 1 annotation out of five with acceptable quality.\n",
    "\n",
    "The can be exported as a tsv file containing an image identifier and a json with markup and some other parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DBrain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A platform in the alpha testing stage.\n",
    "\n",
    "Allows to make segmentation markup with polygons. Has a large pool of annotators, each has a rating based on quality and speed. The platform also claims to monitor the behavior of annotators: whether they take breaks or not, etc.\n",
    "\n",
    "It claims own validation algorithms through consensus but does not disclose the details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Annotation description example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The annotation description should be a single text file, with links to supplementary materials, such as videos and examples.\n",
    "\n",
    "The file structure should correspond to the structure of the \"Annotation description\" section. Text description and tables should be palced in the corresponding subsections of the file; other materials should be available via links to the files:\n",
    "\n",
    "\n",
    "* Annotation dictionary and recommendations should be arranged as two tables.\n",
    "* Annotation instructions should be arranged as a subsection of the description. It may also contain a link to a video file stored in the same folder.\n",
    "* Examples of good and bad markup should be stored in a separate directory, the description should provide a link to this directory.\n",
    "* Annotation format should be described in detail in the appropriate section of the file. The description should be sufficient enough for a new analyst to write a parser on his own. You can also attach a link to the parsing script.\n",
    "* The number of annotators should also be specified in corresponding section of the file. You should specify the total number of annotators *K*, the number of annotators per image *L* and time *t* required for one annotator to process one image.\n",
    "* Description of validation procedure should include: the selected validation method(s), metrics and criteria used to differ good and bad markup. You should also attach a link to the code used for validation.\n",
    "\n",
    "The annotation description, along with all the materials, dataset description and dataset itself, must be archived and stored and distributed as archive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example structure of the annotation description:\n",
    "\n",
    "```\n",
    "Annotation_%dataset_name%/\n",
    "|--- Description.doc\n",
    "|--- Instructions.avi\n",
    "|--- Examples\n",
    "|    |--- Good\n",
    "|    |    |--- Example1 \n",
    "|    |    |--- Example2\n",
    "|    |--- Bad\n",
    "|    |    |--- Example1\n",
    "|    |    |--- Example2\n",
    "|--- Annotation_parser.py\n",
    "|--- Validation_script.py\n",
    "|--- Annotation_tool_configuration\n",
    "|--- ...\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}