Code

Tarfiles#27

Code

Open

bganglia wants to merge 54 commits intomlmed:mastermlmed/torchxrayvision:masterfrom bganglia:tarfilesbganglia/torchxrayvision:tarfilesCopy head branch name to clipboard

+1,453-54

Commits

Commits on Aug 6, 2020

• Open MIMIC from tarfile

  bganglia

  committed
  9f3eb7b

  Copy full SHA for 9f3eb7b

• Merge branch 'master' of https://github.com/ieee8023/torchxrayvision into tarfiles

  bganglia

  committed
  0414355

  Copy full SHA for 0414355

• Merge branch 'master' of https://github.com/ieee8023/torchxrayvision into tarfiles

  bganglia

  committed
  289747d

  Copy full SHA for 289747d

Commits on Aug 8, 2020

• revert whitespace

  bganglia

  committed
  303832c

  Copy full SHA for 303832c

• don't use get_image() in NIH_Dataset

  bganglia

  committed
  da2490b

  Copy full SHA for da2490b

• NIH_Dataset extends TarDataset

  bganglia

  committed
  8fb3f03

  Copy full SHA for 8fb3f03

• Store tarfiles in dictionary

  bganglia

  committed
  395e5e4

  Copy full SHA for 395e5e4

• use getnames intead of getmembers

  bganglia

  committed
  fa69973

  Copy full SHA for fa69973

Commits on Aug 9, 2020

• use O(n) method for determining imgid from tar_path

  bganglia

  committed
  abbbfec

  Copy full SHA for abbbfec

• random data in MIMIC format

  bganglia

  committed
  2ba6f5d

  Copy full SHA for 2ba6f5d

• script for generating random MIMIC data

  bganglia

  committed
  cacc3ad

  Copy full SHA for cacc3ad

• track random MIMIC data

  bganglia

  committed
  ecbf302

  Copy full SHA for ecbf302

• tarfile test using random MIMIC data

  bganglia

  committed
  04f1a32

  Copy full SHA for 04f1a32

• fix test directory

  bganglia

  committed
  90129ab

  Copy full SHA for 90129ab

• use .close() on tarfile and regenerate test directory

  bganglia

  committed
  0aa52a7

  Copy full SHA for 0aa52a7

• support for tarfiles in NIH dataset

  bganglia

  committed
  349babb

  Copy full SHA for 349babb

Commits on Aug 10, 2020

• Inherit from TarDataset in PC_Dataset

  bganglia

  committed
  6999bd3

  Copy full SHA for 6999bd3

• Storage-agnostic dataset

  bganglia

  committed
  842ddf8

  Copy full SHA for 842ddf8

• Inherit from storage agnostic loader

  bganglia

  committed
  37afa4e

  Copy full SHA for 37afa4e

• tidy up tarfile code

  bganglia

  committed
  bbd4007

  Copy full SHA for bbd4007

• remove previous TarDataset, ZipDataset classes

  bganglia

  committed
  34daddb

  Copy full SHA for 34daddb

Commits on Aug 13, 2020

• Scripts for generating test data

  bganglia

  committed
  727d9ff

  Copy full SHA for 727d9ff

• Test data

  bganglia

  committed
  d2ae7c0

  Copy full SHA for d2ae7c0

• Tests for zip, tar in MIMIC, NIH, and PC

  bganglia

  committed
  41b50c4

  Copy full SHA for 41b50c4

• clean up storage classes

  bganglia

  committed
  48d8170

  Copy full SHA for 48d8170

Commits on Aug 26, 2020

• save progress

  bganglia

  committed
  5c4117e

  Copy full SHA for 5c4117e

• inherit from Dataset in NIH_Dataset

  bganglia

  committed
  2773c69

  Copy full SHA for 2773c69

• Add code for automated tests with script-generated data

  bganglia

  committed
  7ffc252

  Copy full SHA for 7ffc252

• script for writing random data

  bganglia

  committed
  68a71ae

  Copy full SHA for 68a71ae

• fall back on .index() instead of trying to load a cached version in .load_dataset()

  bganglia

  committed
  ec9777b

  Copy full SHA for ec9777b

Commits on Aug 27, 2020

• support multiprocessing

  bganglia

  committed
  29498a6

  Copy full SHA for 29498a6

• Clean up new code for tests and format interfaces

  bganglia

  committed
  3674357

  Copy full SHA for 3674357

• write partial metadata files with subset of columns

  bganglia

  committed
  ccec9ae

  Copy full SHA for ccec9ae

• Improve caching

  bganglia

  committed
  c091734

  Copy full SHA for c091734

Commits on Aug 28, 2020

• fix tests

  bganglia

  committed
  e56a565

  Copy full SHA for e56a565

• fix error in data-generation script

  bganglia

  committed
  1dde4b7

  Copy full SHA for 1dde4b7

• create .torchxrayvision if it does not already exist

  bganglia

  committed
  1628db4

  Copy full SHA for 1628db4

• fix line adding .torchxrayvision

  bganglia

  committed
  124467c

  Copy full SHA for 124467c

• Commit sample data for testing NLM_TB datasets, instead of auto-generating it

  bganglia

  committed
  28816e5

  Copy full SHA for 28816e5

• Commit covid test cases

  bganglia

  committed
  ce38e57

  Copy full SHA for ce38e57

• Include parallel tests again

  bganglia

  committed
  281935c

  Copy full SHA for 281935c

• trycatch on reading/writing stored_mappings, with disk_unwriteable_out_of_date variable to prevent reading from an out-of-date cache

  bganglia

  committed
  9c2c9d2

  Copy full SHA for 9c2c9d2

• work when .torchxrayvision is not writeable

  bganglia

  committed
  7c6aebb

  Copy full SHA for 7c6aebb

• remove some print statements

  bganglia

  committed
  cb97e70

  Copy full SHA for cb97e70

• add test simulating an unwriteable disk

  bganglia

  committed
  950ae96

  Copy full SHA for 950ae96

• use filesystem instead of dictionary

  bganglia

  committed
  300c9d7

  Copy full SHA for 300c9d7

Commits on Aug 30, 2020

• rewrite data generation scripts as python, not bash scripts; add parameter for length of all datasets

  bganglia

  committed
  218fa75

  Copy full SHA for 218fa75

Commits on Aug 31, 2020

• cleanup: better variable names and use blake2b instead of hash (works between sessions)

  bganglia

  committed
  b22cead

  Copy full SHA for b22cead

• Add test for asserting a dataset loads faster the second time

  bganglia

  committed
  ae09bc9

  Copy full SHA for ae09bc9

• Don't invoke duration test, to avoid spurious errors

  bganglia

  committed
  30c043b

  Copy full SHA for 30c043b

• Call on new data generation script

  bganglia

  committed
  bfdebf2

  Copy full SHA for bfdebf2

Commits on Sep 5, 2020

• simplify and improve documentation

  bganglia

  committed
  0f7ea51

  Copy full SHA for 0f7ea51

Commits on Sep 19, 2020

• reorganize

  bganglia

  committed
  71c7a50

  Copy full SHA for 71c7a50

• Fix path length in CheX_Dataset

  bganglia

  committed
  1715b9d

  Copy full SHA for 1715b9d