TCIA_STL_Visualization_with_itkWidgets.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "You can download and run this notebook locally, or you can run it for free in a cloud environment using Colab or Sagemaker Studio Lab:\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kirbyju/TCIA_Notebooks/blob/main/TCIA_STL_Visualization_with_itkWidgets.ipynb)\n",
        "\n",
        "[![Open In SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github.com/kirbyju/TCIA_Notebooks/blob/main/TCIA_STL_Visualization_with_itkWidgets.ipynb)"
      ],
      "metadata": {
        "id": "sBJRm40-kiGQ"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N92pMd7ig0ZS"
      },
      "source": [
        "# Summary\n",
        "Interactive visualization of data is essential to gaining new insights into problem domains, verifying image processing pipelines, and evaluating experimental results. TCIA provides a vast quantity of high-quality clinical data for such work. itkWidgets provides methods for visualizing TCIA's data within Python notebooks, whether they are running locally or in the cloud.\n",
        "\n",
        "This notebook explores the visualization of \"non-traditional\" data stored in TCIA.  Other notebooks in this repository demonstrate the visualization of DICOM, SEG, and RTStruct objects.  Herein, we demonstrate the visualization of STL files that are commonly used in 3D printing.\n",
        "\n",
        "[The Cancer Imaging Archive (TCIA)](https://www.cancerimagingarchive.net/) is a public service funded by the National Cancer Institute that addresses this challenge by providing hosting and de-identification services to take major burdens of data sharing off researchers. Its rich collection of clinical data and annotations is particularly powerful as a community resource when it is paired with interactive code systems, such as Jupyter systems.\n",
        "\n",
        "While Jupyter systems (JupyterLab, Jupyter Notebook, Google CoLab, Amazon SageMaker, etc.) are outstanding tools for interactive programming and algorithm sharing, the ability to visualize data within them has traditionally be limited to plots and sparse 2D slices, which are insufficient when dealing with 3D clinical data. This limitation is particularly challenging when using cloud-based Jupyter systems such as Google Colab and Amazon SageMaker.  3D interactive visualization of data when using those cloud systems traditionally requires downloading that data to your local machine and then running a separate visualization application. itkWidgets addresses these visualization challenges by providing interacitve 3D (and 2D) visualizations within the cells of your Jupyter scripts.\n",
        "\n",
        "[itkWidgets](https://github.com/InsightSoftwareConsortium/itkwidgets) allows researchers to visualize images, point sets, and 3D geometry in Jupyter systems (Jupyer Notebook, JupyterLab, Amazon SageMaker, and Google Colab). Despite its name, itkWidgets does not require the use of ITK. It can directly visualize numpy arrays, torch tensors, DASK arrays, VTK polydata, and a multitude of other Python data structures.  It is open source and freely available with the Apache 2.0 license.\n",
        "\n",
        "[VTK](https://vtk.org) is a freely available, open-source library for scientific visualization.  It is available in C++, Python, JavaScript, and several other languages.  In this notebook, we include it for its ability to read STL files."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nj6P7YCmll4S",
        "tags": []
      },
      "source": [
        "# Outline\n",
        "\n",
        "1. Setup\n",
        "2. TCIA Basics\n",
        "3. STL (VTK) Basics\n",
        "4. itkWidgets Basics\n",
        "5. Use Cases\n",
        "    1. Visualize STL points\n",
        "    2. Convert STL to a labelmap"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NnWj7CJhCdZL"
      },
      "source": [
        "# 1. Setup\n",
        "\n",
        "These are the initial steps for running notebooks within various Jupyter environments."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tA8E3uJJCdZM"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import sys\n",
        "\n",
        "# Upgrade pip, just in case.\n",
        "!{sys.executable} -m pip install --upgrade -q pip"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mdmx436ICdZN"
      },
      "outputs": [],
      "source": [
        "# If running on SageMaker or Studio Lab, install essential packages and extensions.\n",
        "if \"studio-lab-user\" in os.getcwd():\n",
        "    print(\"Upgrading dependencies\")\n",
        "    !conda install --yes -q --prefix {sys.prefix} -c conda-forge opencv nodejs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NHvGynu7hFCO"
      },
      "source": [
        "# On many systems you must manually install the imjoy-jupyter-extension!!\n",
        "\n",
        "If you do not see a blue 'ImJoy' icon on the menu bar in this notebook:\n",
        "   1) Google CoLab: The following does not apply to Google CoLab - it will not show an ImJoy and all\n",
        "      should work without modification.\n",
        "   2) Enable Extensions:  Many Jupyter Lab systems disable jupyter extensions by default,\n",
        "      and they must be enabled for this notebook to work. Typically this is accomplished using the\n",
        "      Jupyter interface to select the extension manager (left-hand side, icon that looks like a piece\n",
        "      of a puzzle) and select the Enable button if it appears.\n",
        "   2) Install imjoy extension: In the extension manager, search for 'imjoy' and install\n",
        "      the 'imjoy-jupyter-extension'. The installation can take several minutes. It may also prompt\n",
        "      you to rebuild, save, and reload your jupyter environment as part of this process.  In the end,\n",
        "      you should see a blue 'ImJoy' icon on the left side of the menu bar in this notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XFhjBkS35hYN",
        "outputId": "408c7a4d-492f-43cf-cb38-afa92d1f53d3"
      },
      "source": [
        "# 2. TCIA Basics\n",
        "\n",
        "[Browsing Collections](https://www.cancerimagingarchive.net/collections) and viewing [Analysis Results](https://www.cancerimagingarchive.net/tcia-analysis-results/) of TCIA datasets are the easiest ways to become familiar with what is available.  These pages will help you quickly identify datasets of interest, find valuable supporting data that are not available via our APIs (e.g. clinical spreadsheets, non-DICOM segmentation data), and answer the most common questions you might have about the datasets.  \n",
        "\n",
        "If you are new to accessing TCIA via notebooks, you can find additional tutorials on querying and downloading data at https://github.com/kirbyju/TCIA_Notebooks. We will also leverage [tcia_utils](https://github.com/kirbyju/tcia_utils) in this notebook to make it easier to grab our data to visualize."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9ziuezFDCdZO"
      },
      "outputs": [],
      "source": [
        "# Install tcia_utils\n",
        "!{sys.executable} -m pip install --upgrade -q tcia_utils"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tF8-MqoN6YT5"
      },
      "outputs": [],
      "source": [
        "from tcia_utils import nbia\n",
        "import pandas as pd\n",
        "import requests"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "urhgDgvb39pS"
      },
      "outputs": [],
      "source": [
        "# Download a \"Shared Cart\" that has been previously\n",
        "#    created via NBIA.  It contains the DICOM image objects.\n",
        "#    This is case Prostate-MRI-US-Biopsy-0001.  It has MR and US acquisitions.\n",
        "cartName = \"nbia-63501667763872140\"\n",
        "\n",
        "# retrieve cart metadata\n",
        "cart_data = nbia.getSharedCart(cartName)\n",
        "\n",
        "# download the series_uids list and return dataframe of metadata\n",
        "df = nbia.downloadSeries(cart_data, format = \"df\")\n",
        "\n",
        "# display dataframe\n",
        "display(df)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ljTswwi-hFCQ"
      },
      "outputs": [],
      "source": [
        "# For this demo...\n",
        "\n",
        "# Install itk for DICOM I/O and for reading DICOM into an itkImage\n",
        "#   that manages all DICOM field values, include acquistion details\n",
        "#   such as voxel image, image orientation, and image directions,\n",
        "#   which are critical to image processing and display.\n",
        "!{sys.executable} -m pip install --upgrade -q \"itk==5.4.0\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KQ4weJ9xhFCR"
      },
      "outputs": [],
      "source": [
        "import glob\n",
        "\n",
        "import itk\n",
        "\n",
        "dicom_data_dir = \"tciaDownload\"\n",
        "\n",
        "# The series_uid defines their directory where the MR data was stored on disk.\n",
        "mr_series_uid = df.at[df.Modality.eq('MR').idxmax(), 'Series UID']\n",
        "dicom_mr_dir = os.path.join(dicom_data_dir, mr_series_uid)\n",
        "\n",
        "# The series_uid defines their directory where the US data was stored on disk.\n",
        "# The ultrasound data is stored as a single 3D DICOM object, so we resolve its filename using glob\n",
        "us_series_uid = df.at[df.Modality.eq('US').idxmax(), 'Series UID']\n",
        "dicom_us_dir = os.path.join(dicom_data_dir, us_series_uid)\n",
        "dicom_us_file = glob.glob(os.path.join(dicom_us_dir, \"*.dcm\"))[0]\n",
        "\n",
        "# Load and sort the DICOM data into a volume - since no series_uid is\n",
        "#    specified with this read command, it will load the first series in the directory.\n",
        "dicom_mr_image = itk.imread(dicom_mr_dir, itk.F)\n",
        "dicom_us_image = itk.imread(dicom_us_file, itk.F)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_b24pK9ACdZQ"
      },
      "outputs": [],
      "source": [
        "# Download the STL data from TCIA.  STL files are typically stored as a collection\n",
        "#     in a zipped file.  For example, for the Prostate-MRI-US-Biopsy, the STL data\n",
        "#     for every patient is stored in a single 247 MB zip file.  Rather than spend time\n",
        "#     downloading and then uncompressing that data for this demo, we will use a single STL\n",
        "#     file stored on GitHub.\n",
        "\n",
        "stl_filename = \"Prostate-MRI-US-Biopsy-0001-ProstateSurface-seriesUID-1.3.6.1.4.1.14519.5.2.1.266717969984343981963002258381778490221.STL\"\n",
        "stl_url = \"https://github.com/kirbyju/TCIA_Notebooks/raw/main/data/\" + stl_filename\n",
        "\n",
        "stl_data_filename = os.path.join(dicom_data_dir, stl_filename)\n",
        "\n",
        "request = requests.get( stl_url )\n",
        "open(stl_data_filename, \"wb\").write(request.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UWeCJUcXCdZQ",
        "tags": []
      },
      "source": [
        "# 3. STL (VTK) Basics\n",
        "\n",
        "[STL](https://en.wikipedia.org/wiki/STL_(file_format)) files describe the surfaces of objects, typically for computer-aided design, 3D printing, and computer-aided manufacturing.  In the medical field, they have seen applications ranging from 3D printing representations of organs and complex anatomical structures for surgical planning, to 3D printing of research into cellular scaffords and artificial organs for transplantation.\n",
        "\n",
        "We will use VTK for loading STL files that have been downloaded from TCIA."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zFD9nh2xCdZQ"
      },
      "outputs": [],
      "source": [
        "# Install VTK\n",
        "!{sys.executable} -m pip install --upgrade -q vtk"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KajnXtvbCdZR"
      },
      "outputs": [],
      "source": [
        "import vtk\n",
        "\n",
        "# Read the STL file from local disk\n",
        "reader = vtk.vtkSTLReader()\n",
        "reader.SetFileName(stl_data_filename)\n",
        "reader.Update()\n",
        "stl_polydata = reader.GetOutput()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B9njZ1_JCdZR",
        "tags": []
      },
      "source": [
        "# 4. itkWidget Basics\n",
        "\n",
        "[itkWidgets documentation](https://itkwidgets.readthedocs.io/en/latest/?badge=latest) provides a summary and illustrations of itkWidgets for a wide variety of scientific data visualization use cases.  Here we focus on its application to data on TCIA."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GfTy1b-GCdZR"
      },
      "outputs": [],
      "source": [
        "# This is the installation required for itkWidgets\n",
        "!{sys.executable} -m pip install --upgrade -q \"itkwidgets[all]==1.0a53\" imjoy_elfinder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lzsGydMXCdZS"
      },
      "outputs": [],
      "source": [
        "# This is the most common import command for itkWidgets.\n",
        "#   The view() function opens an interactive viewer for 2D and 3D\n",
        "#   data in a variety of formats.\n",
        "from itkwidgets import view"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jYuHATZICdZS",
        "tags": []
      },
      "source": [
        "# 5. Use Cases\n",
        "\n",
        "5.A. Visualize STL Points<br>\n",
        "5.B. Convert STL to Labelmap<br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mgsjf4xR5SlI"
      },
      "source": [
        "## 5.A. Visualize STL Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8eUpuvOfCdZS"
      },
      "outputs": [],
      "source": [
        "# View the STL data as a point cloud using itkWidgets, to verify the data was read correctly.\n",
        "\n",
        "viewerA = view(point_sets=stl_polydata)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wEHLB31cCdZS"
      },
      "source": [
        " ## 5.B. Convert STL to Labelmap"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gJmaRr01CdZT"
      },
      "outputs": [],
      "source": [
        "fg_image = itk.Image[itk.F,3].New()\n",
        "fg_image.SetRegions(dicom_mr_image.GetLargestPossibleRegion())\n",
        "fg_image.CopyInformation(dicom_mr_image)\n",
        "fg_image.Allocate()\n",
        "fg_image.FillBuffer(1)\n",
        "fg_vtk_image = itk.vtk_image_from_image(fg_image)\n",
        "\n",
        "# This creates a foreground image that defines the extent and template for rendering the STL data.\n",
        "\n",
        "# The above command works because the dicom_mr_image has an identity matrix for its directions.\n",
        "# Regretfully, vtk doesn't support images with non-identify acquisition directions, so be careful."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XCDE8KGoCdZT"
      },
      "outputs": [],
      "source": [
        "# polygonal data -> image stencil:\n",
        "pol2stenc = vtk.vtkPolyDataToImageStencil()\n",
        "pol2stenc.SetInputData(stl_polydata)\n",
        "pol2stenc.SetOutputOrigin(dicom_mr_image.GetOrigin())\n",
        "pol2stenc.SetOutputSpacing(dicom_mr_image.GetSpacing())\n",
        "pol2stenc.SetOutputWholeExtent(fg_vtk_image.GetExtent())\n",
        "pol2stenc.Update()\n",
        "\n",
        "# image stencil -> image:\n",
        "imgstenc = vtk.vtkImageStencil()\n",
        "imgstenc.SetInputData(fg_vtk_image)\n",
        "imgstenc.SetStencilConnection(pol2stenc.GetOutputPort())\n",
        "imgstenc.ReverseStencilOff()\n",
        "imgstenc.SetBackgroundValue(0)\n",
        "imgstenc.Update()\n",
        "\n",
        "stl_vtk_contour_image = imgstenc.GetOutput()\n",
        "\n",
        "stl_contour_image = itk.image_from_vtk_image(stl_vtk_contour_image)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OntbqeAHCdZT"
      },
      "outputs": [],
      "source": [
        "viewerB = view(image=dicom_mr_image, label_image=stl_contour_image)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "C6qFIu3ChFCT"
      },
      "outputs": [],
      "source": [
        "viewerB.set_image_color_map(\"Grayscale\")\n",
        "viewerB.set_image_color_range([0,300])\n",
        "viewerB.set_view_mode(\"ZPlane\")\n",
        "viewerB.set_ui_collapsed(False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DYXsNGcY93B8",
        "tags": []
      },
      "source": [
        "# Acknowledgements\n",
        "\n",
        "TCIA is funded by the [Cancer Imaging Program (CIP)](https://imaging.cancer.gov/), a part of the United States [National Cancer Institute (NCI)](https://www.cancer.gov/), and is managed by the [Frederick National Laboratory for Cancer Research (FNLCR)](https://frederick.cancer.gov/).\n",
        "\n",
        "If you leverage this notebook or any TCIA datasets in your work, please be sure to comply with the [TCIA Data Usage Policy](https://wiki.cancerimagingarchive.net/x/c4hF). In particular, make sure to cite the DOI(s) for the specific TCIA datasets you used in addition to TCIA citation provided below!\n",
        "\n",
        "This notebook was created by [Stephen Aylward (Kitware)](https://www.kitware.com/stephen-aylward/), [Justin Kirby (Frederick National Laboratory for Cancer Research)](https://www.linkedin.com/in/justinkirby82/), [Brianna Major (Kitware)](https://www.kitware.com/brianna-major/), and [Matt McCormick (Kitware)](https://www.kitware.com/matt-mccormick/).   The creation of this notebook was funded, in part, by NIBIB and NIGMS R01EB021396, NIBIB R01EB014955, NCI R01CA220681, and NINDS R42NS086295.\n",
        "\n",
        "If you have any questions, suggestions, or issues with itkWidgets, please post them on the [itkwidget issue tracker](https://github.com/InsightSoftwareConsortium/itkwidgets/issues) or feel free to email us at kitware@kitware.com.\n",
        "\n",
        "## Data Citation\n",
        "The data used in this notebook is part of the Prostate-MRI-US-Biopsy collection:\n",
        "\n",
        "Natarajan, S., Priester, A., Margolis, D., Huang, J., & Marks, L. (2020). Prostate MRI and Ultrasound With Pathology and Coordinates of Tracked Biopsy (Prostate-MRI-US-Biopsy) [Data set]. The Cancer Imaging Archive. DOI: 10.7937/TCIA.2020.A61IOC1A\n",
        "\n",
        "## Publication Citation\n",
        "The data used in this notebook is part of the Prostate-MRI-US-Biopsy collection:\n",
        "\n",
        "Sonn GA, Natarajan S, Margolis DJ, MacAiran M, Lieu P, Huang J, Dorey FJ, Marks LS. Targeted biopsy in the detection of prostate  cancer using an office based magnetic resonance ultrasound fusion device.  Journal of Urology 189, no. 1 (2013): 86-91. DOI: 10.1016/j.juro.2012.08.095\n",
        "\n",
        "## TCIA Citation\n",
        "\n",
        "Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. Journal of Digital Imaging, 26(6), 1045–1057. https://doi.org/10.1007/s10278-013-9622-7"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "medical-image-ai:Python",
      "language": "python",
      "name": "conda-env-medical-image-ai-py"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.15"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}