2022-03-21.md

2022-03-21

Agenda

• DICOMweb performance

  • multi-part mime vs single part for video
  • Is single part API new?
    • Redocument effort changed how bulk data should work
      • Original documentation - it was multi-part mime wrapped
      • New documentation added support for single part
        • Single part vs multi part controlled via content-type header
    • CORS can be bypassed if you specify the content-type without using quotes (")
      • Still an issue if you use authorization HTTP Headers
        • Passing token via URI or cookie is OK though
      • Documentation on this here: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#simple_requests
      • Example commit: https://github.com/cornerstonejs/cornerstoneWADOImageLoader/commit/37b076d336951d3added50b352c4f87b78778f25
  • What is the fastest way to download entire study in DICOM P10 format from DICOMweb
    • Some people are calling the study or series multi-part mime API /studies/studyuid /studies/studyuid/series/seriesuid
    • This is not ideal for the following reasons:
      • The maximum download rate over a single connection is limited. (e.g. 100 Gbps, you might get 2 Gbps with this method)
        • TCP is chatty and this is noticeable at high bandwidths
        • TCP Window Size Limitations/issues configuring
      • Server needs to "build" the multi-part mime response
        • should be able to stream it
        • can use a lot of memory if your not careful (need to retrieve several in parallel but not all)
      • Client needs to "parse" the multi-part mime response
        • Requires lots of things to go right to avoid massive delays and memory usage
        • Not allow HTTP Body to be accessed until entire response read
        • Client environment limitations on buffer sizes (e.g. browsers limit arrays to 4GB RAM)
        • To do this right:
          • Need stream based network API
          • Need stream based multi-part mime parser
          • Need code that can make sense of streamed instances
    • Better to use HTTP/2 and make concurrent requests
      • Requires a smart DICOMweb implementation
        • DCM4CHEE / frame endpoint is slow at extracting individual frames from a DICOM P10 instance
          • Bill has a test digital pathology image with 32,000 frames in one SOP Instance - takes 60 seconds on DCM4CHEE
            • Can probably find data like this in the IDC: https://datacommons.cancer.gov/repository/imaging-data-commons
          • This could be resolved by storing indexes of each frame to avoid seeking from beginning each time
    • Speed mainluy depends on how the service was implemented
      • http://chafey.blogspot.com/2015/04/making-dicomweb-fast.html

• DICOMweb WG27 Activities

  • Relative URI CP
    • Standard as written does not say you can't use relative URIs, so you have to fall back on the standard it references to understand what it means
    • This CP defines this explicitly
      • All metadata requests are relative to the study root (e.g. studies/$studyuid)
  • JPEG-XL Work Item
    • Targeting having the supplement ready in 3 months
    • JPEG-XL vs HTJ2K performance: https://docs.google.com/document/d/1gS3cc7p00M3_Ub-0AC7shm16H_FCC2IyxqE2TCjpzUI/edit
    • Main value is enhanced color support (which is limited in all other codecs except JPEG2000)
    • NOTE: There are well defined ways of doing the color transform in the browser via SVG, but only supported for 8 bit data

• DICOM QIDO-RS Study Query in memory project

  • https://github.com/chafey/qido-rs-in-memory-js
  • loaded 1M studies into in memory db https://github.com/techfort/LokiJS
  • 50MB for 1M Studies (JSON size)
  • Question: how to keep it up to date with changes in data store?
    • Need to find the right in memory db technology
  • Idea: duckdb?: https://duckdb.org/
    • https://duckdb.org/2021/12/03/duck-arrow.html

• DICOM Test Data Generator

  • JF Created his own
  • Simon created his own...
  • Need a "Synthia for FHIR" for DICOM
  • Pantelis DICOM WSI Generator: https://gist.github.com/PantelisGeorgiadis/fb9e305610c92633fd93fcc3e973d8a2
  • Synthia FHIR ImagingStudy Generator: https://github.com/synthetichealth/synthea/wiki/Generic-Module-Framework%3A-States#imagingstudy

• MIMIC data sets

  • 400k chest radiographs released as DICOM. Some tags anonymized, some scrubbed
  • Requires data use agreement
  • https://physionet.org/content/mimic-cxr/2.0.0/
  • MIMIC has other non imaging data too - EHR?