Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support downloading old versions of objects #36

Merged
merged 14 commits into from
Nov 25, 2024
Merged

Support downloading old versions of objects #36

merged 14 commits into from
Nov 25, 2024

Conversation

jwodder
Copy link
Member

@jwodder jwodder commented Nov 21, 2024 •
edited
Loading

Closes #9.
Closes #10.
Closes #12.
Closes #39.

@jwodder jwodder self-assigned this Nov 21, 2024
@jwodder jwodder marked this pull request as ready for review November 25, 2024 15:47
@jwodder
Copy link
Member Author

jwodder commented Nov 25, 2024

@yarikoptic Progress report: Using commit f4df37a from this PR, I ran:

cargo run -r -- \
    --path-filter '^zarr-checksums/771f886c-8c2c-4fd5-afdc-3821d4043977' \
    s3://dandiarchive/dandiarchive/dandiarchive/ \
    dump

(where the given zarr-checksums/... path appears at the start of one of the first CSV lists for the most recent manifest) and, in under five(?) minutes, the following tree had been downloaded:

dump/
└── zarr-checksums/
    └── 771f886c-8c2c-4fd5-afdc-3821d4043977/
        ├── 1/
        │   └── 0/
        │       └── 0/
        │           └── 7/
        │               ├── 2/
        │               │   ├── .checksum.old.Y5YOjDSUbchNsT.iyfRMITFG0v2rKuUg.3b51c805def85a25d5e4ba12c9442a15
        │               │   ├── .checksum.old.bZiYjk5XdD4nW2UazK.S_RztDs5XRVVo.3b51c805def85a25d5e4ba12c9442a15
        │               │   ├── .checksum.old.yz9RemIYL3zPg1OlAQWmVG6zqCYzU8FR.3b51c805def85a25d5e4ba12c9442a15
        │               │   └── .checksum.old.zSmYc2bmUCfYBMMR1iJHa4z1YAnUhfqC.3b51c805def85a25d5e4ba12c9442a15
        │               ├── 3/
        │               │   ├── .checksum
        │               │   ├── .checksum.old.4Cw3.K1V5ZAbUQGNN2YMcwsx.kVT3OHW.ca9f26c5e317a224dcfc3f4a01120368
        │               │   ├── .checksum.old.Jv6Eq7FvtbufG4vzzQCWF07tk6.8cofM.ca9f26c5e317a224dcfc3f4a01120368
        │               │   ├── .checksum.old.WYHtPHHRzLiQLckh0LgbyytijzmvbO1r.ca9f26c5e317a224dcfc3f4a01120368
        │               │   ├── .checksum.old.hYtJOGWaM4HSLo3KdrNTl6TTrXs9bd1Y.ca9f26c5e317a224dcfc3f4a01120368
        │               │   └── .s3invsync.versions.json
        │               ├── 4/
        │               │   ├── .checksum
        │               │   ├── .checksum.old.PYPtFBJVpjcyiCL3yHsnYwq18mdKYaqm.5e3a65297adafb6e52183c9933d5e128
        │               │   ├── .checksum.old.Xm10v2X6raSQziXOABtktlbKqllFEQiS.5e3a65297adafb6e52183c9933d5e128
        │               │   ├── .checksum.old.sA1zbdcxb7.KCnUHZBwSfBZbM6BJ33sj.5e3a65297adafb6e52183c9933d5e128
        │               │   ├── .checksum.old.xdhnPcm4RmcWYYEFIqQYJWebSRpygFmc.5e3a65297adafb6e52183c9933d5e128
        │               │   └── .s3invsync.versions.json
        │               ├── 5/
        │               │   ├── .checksum
        │               │   ├── .checksum.old.MfcJXZccOJxK1j5JWDH4IQAW_uuSoQs_.6379510970f5e2b329259f636991bbab
        │               │   ├── .checksum.old._1IS4jeOi5ST72YMJ1Fb4b_fXQ9itFtV.6379510970f5e2b329259f636991bbab
        │               │   ├── .checksum.old.kynOdZ9MVLDY3sT861hpmGiRwOAhycJF.6379510970f5e2b329259f636991bbab
        │               │   ├── .checksum.old.mXq0x58nXRHGqxoXPq96YiRYWHHnQbuK.6379510970f5e2b329259f636991bbab
        │               │   └── .s3invsync.versions.json
        │               ├── 6/
        │               │   ├── .checksum
        │               │   ├── .checksum.old.5rHBg8m1goFAJtQRJg.oK5ns574h360R.c1107a21726ff2aa58a8f162be407324
        │               │   ├── .checksum.old.BGAo.OUbBRL64kBSG0nhF_Bpj8I0nCSN.c1107a21726ff2aa58a8f162be407324
        │               │   ├── .checksum.old.G5rxcXaNOvEey8A5ZWeHRE7mFUKNejUm.c1107a21726ff2aa58a8f162be407324
        │               │   ├── .checksum.old.OLZrML_2a_zDpmKzJR8CIS1Sre8xCqOE.434940eee1e1172be9aa8c0b3ec709d6
        │               │   ├── .checksum.old.Rjixgoi5VGxoQPzdOugsC3pSTWGa.t5j.c1107a21726ff2aa58a8f162be407324
        │               │   ├── .checksum.old.YEI0qPX6jg4mxvIBl5E5yeLcrEuumn_j.c1107a21726ff2aa58a8f162be407324
        │               │   ├── .checksum.old._QMvI_QYKXgUPiVrzXTo_N8VaeqG7L9g.434940eee1e1172be9aa8c0b3ec709d6
        │               │   ├── .checksum.old.bsJqPTpVaQb4hj9KrI8BXLi2eyWMq7nh.434940eee1e1172be9aa8c0b3ec709d6
        │               │   ├── .checksum.old.gnBxor3S71Ec5vHjI3RDjegs.9UjhT9q.434940eee1e1172be9aa8c0b3ec709d6
        │               │   └── .s3invsync.versions.json
        │               └── 7/
        │                   ├── .checksum
        │                   ├── .checksum.old.J18cn1gf6YbR09iwypocDcN_4ycYzc2u.a7093da07c6cea0cd368274099aa3d99
        │                   ├── .checksum.old.WfWG7RFKnAOXklcKEMaepKKCqiHPUIv9.a7093da07c6cea0cd368274099aa3d99
        │                   ├── .checksum.old.dAMIR8dngbAuOUpP_Jm3UhtZZiNC2XIy.a7093da07c6cea0cd368274099aa3d99
        │                   ├── .checksum.old.opIgnXhDFo3U.VG9.Bqz7AQfs3P6HtYq.a7093da07c6cea0cd368274099aa3d99
        │                   └── .s3invsync.versions.json
        ├── 2/
        │   ├── .checksum
        │   ├── .checksum.old.2sze.6f96inhepq1HpFmAMW92uFsHcGr.0156573b77c2c5d3badc5fd6e6d0707f
        │   ├── .checksum.old.525l6VvTIxcrkktU1cKHDj84LA7ksNm3.65771ffc304285991f79a5b851ccbed6
        │   ├── .checksum.old.86birFGM3YCUJO7dXxhOPyKa8av_LZ70.65771ffc304285991f79a5b851ccbed6
        │   ├── .checksum.old.AbgJzJmZZfrUr36ge4s0jrs7Nnjwki3H.0156573b77c2c5d3badc5fd6e6d0707f
        │   ├── .checksum.old.GIJ09B0_AYQBkOZmy9JtLgMx5YKkqhcl.65771ffc304285991f79a5b851ccbed6
        │   ├── .checksum.old.h2d1zW.TsDqHIzvkcxv9MKlQleWAhW1r.0156573b77c2c5d3badc5fd6e6d0707f
        │   ├── .checksum.old.pkPKHF1DB9AIE6Hu_D0LsOfuuOXmKGKe.65771ffc304285991f79a5b851ccbed6
        │   ├── .checksum.old.uECEkgeMswyeTp4hd5h6POPaYm8Q79RR.0156573b77c2c5d3badc5fd6e6d0707f
        │   ├── .checksum.old.wUtSmxOdHj.4oLfmcpvgi9yDrUF.xudL.0156573b77c2c5d3badc5fd6e6d0707f
        │   ├── .s3invsync.versions.json
        │   └── 0/
        │       ├── .checksum
        │       ├── .checksum.old.41IBWri8IgQTy3wpkLSnth6r.LGmrXUP.20e01df403ee6d675dfd276f4d8f256c
        │       ├── .checksum.old.4KkJaDIvcZNIjx2U0TGPr4e3heroqNfc.e195f068c33e5a63372b7fbe0b282ea7
        │       ├── .checksum.old.NRCWuwq0npFyl0iHmswIXqkLURJ2Tk_u.e195f068c33e5a63372b7fbe0b282ea7
        │       ├── .checksum.old.PHJq0J1PRfHeckot4lYjUwZNz6PZ7Ykb.20e01df403ee6d675dfd276f4d8f256c
        │       ├── .checksum.old.Wa_Q68EDPePad.tpruPllzjCccgpg06w.e195f068c33e5a63372b7fbe0b282ea7
        │       ├── .checksum.old.ZuWJTbkEMFd6VCM2zx.SqvTjpIUxH9cO.e195f068c33e5a63372b7fbe0b282ea7
        │       ├── .checksum.old.gHXZWkrUUmpq3MwI2HqCRvy3L1xYO2eO.20e01df403ee6d675dfd276f4d8f256c
        │       ├── .checksum.old.mw3VmdzS.cCDKvesFtSv24SPLy0eslNJ.20e01df403ee6d675dfd276f4d8f256c
        │       ├── .checksum.old.sf1NLO1DmhMYl5EvLLl09JxXIT6_4bXx.e195f068c33e5a63372b7fbe0b282ea7
        │       ├── .s3invsync.versions.json
        │       └── 0/
        │           ├── .checksum
        │           ├── .checksum.old.4cvCyqjPsu.RHSgubz8EFJnFUpHBbnrM.cd8a34cdc3783f0c2dfa1531b9908239
        │           ├── .checksum.old.Hf3ujWdZMosyIN6fcalWedDBMmJa_EkL.cd8a34cdc3783f0c2dfa1531b9908239
        │           ├── .checksum.old.MQxgC6dv07wyKJTN6Dyotyf8Q0iyvCYs.f0b196a52c015935b940f547fa4768a9
        │           ├── .checksum.old.Rh8v9tKWPxsUWc_mJBAMzjn0gXQpf_hT.cd8a34cdc3783f0c2dfa1531b9908239
        │           ├── .checksum.old.bemZiqrLnbm_dFl7GFl2eC86azM38tzo.cd8a34cdc3783f0c2dfa1531b9908239
        │           ├── .checksum.old.fSDoGFFN5ELa9KtcJhZR5VVnHsyvXE_k.f0b196a52c015935b940f547fa4768a9
        │           ├── .checksum.old.hxTwgdoBSuxGiIS3iQikxJLmXC0CN7u2.f0b196a52c015935b940f547fa4768a9
        │           ├── .checksum.old.qk.Im0y1eiixJDMayHqOoJrwr8yzgkHJ.cd8a34cdc3783f0c2dfa1531b9908239
        │           ├── .checksum.old.zkMeJ.X4QmP_u2v.esOZg81Z9.tdQ9Ed.f0b196a52c015935b940f547fa4768a9
        │           ├── .s3invsync.versions.json
        │           ├── 0/
        │           │   ├── .checksum
        │           │   ├── .checksum.old.6iPmPaJ53EUBP5VqwD7fQbnMszxcntfx.e150df6ae1cfcdb4a8144af31f55ae39
        │           │   ├── .checksum.old.N5GNeJokuxmVrh5jRCdvJQOz3eCiRf4i.e150df6ae1cfcdb4a8144af31f55ae39
        │           │   ├── .checksum.old.TUqUW5CO5ZbmGJfoV1ZrznUMb41f1.xj.e150df6ae1cfcdb4a8144af31f55ae39
        │           │   ├── .checksum.old.Tuy2l30MAuWJKnmc18vZmSelfZkHvKrG.e150df6ae1cfcdb4a8144af31f55ae39
        │           │   ├── .s3invsync.versions.json
        │           │   ├── 0/
        │           │   │   ├── .checksum
        │           │   │   ├── .checksum.old.CmcjkaP0zr2JfWFznTMy85GP2MQzLTUa.b5f7f65ff55607ff111eeead7fca1cce
        │           │   │   ├── .checksum.old.DOi9JFRKkRF0SmVn4nhne7zv6dpy2Zqd.b5f7f65ff55607ff111eeead7fca1cce
        │           │   │   ├── .checksum.old.JD3DLX2WkiUHLytcjsa1mpFwRGwFgGzW.b5f7f65ff55607ff111eeead7fca1cce
        │           │   │   ├── .checksum.old.KDnj.nabh17TArLv3tnEC3mgGDGlBF2H.b5f7f65ff55607ff111eeead7fca1cce
        │           │   │   └── .s3invsync.versions.json
        │           │   ├── 1/
        │           │   │   ├── .checksum
        │           │   │   ├── .checksum.old.DOi8Q6ojKTvKsl57KhKahLDvojkKpQEn.bfb4df8e9712a207728b81b8abcfe55a
        │           │   │   ├── .checksum.old.FiuJM4_z9kkKaG3eO4FNVUf4GuFd.DWK.bfb4df8e9712a207728b81b8abcfe55a
        │           │   │   ├── .checksum.old.S5g9baJjj71h0BYHww9nkXXCxyxxrFoH.bfb4df8e9712a207728b81b8abcfe55a
        │           │   │   ├── .checksum.old.ezcGgloUULgiHhtoh7tplLmD8.24eyWv.bfb4df8e9712a207728b81b8abcfe55a
        │           │   │   └── .s3invsync.versions.json
        │           │   ├── 2/
        │           │   │   ├── .checksum
        │           │   │   ├── .checksum.old.7haRXVAoQ1DE681DmBe7SKiSJMCUrcnP.fcf724d0c8052dcc623b0776919c9e38
        │           │   │   ├── .checksum.old.82bbyAh97kVfKONmRDo6GenEQatHeDVO.fcf724d0c8052dcc623b0776919c9e38
        │           │   │   ├── .checksum.old.L_swoh3m9xXuNVzKQuwqrA7s942pHCxr.fcf724d0c8052dcc623b0776919c9e38
        │           │   │   ├── .checksum.old.YCeurNywZGneu05andUJ7KThV0RWUnXO.fcf724d0c8052dcc623b0776919c9e38
        │           │   │   └── .s3invsync.versions.json
        │           │   └── 3/
        │           │       ├── .checksum
        │           │       ├── .checksum.old..wA75UpeMpK2lv9VX4kmVUOGTz.PAEhp.e877afc6ee68cc8c9c0fa9db14061e73
        │           │       ├── .checksum.old.3EoGueHQ.uBMGVMuGhmel8Uvf5AVoI7U.e877afc6ee68cc8c9c0fa9db14061e73
        │           │       ├── .checksum.old.ICVTtQXrcs.GyHhr1EucFkEEy2luCOQu.e877afc6ee68cc8c9c0fa9db14061e73
        │           │       ├── .checksum.old.a4pNJzz27WXR8fHjHdGzOttSM37tzjzS.e877afc6ee68cc8c9c0fa9db14061e73
        │           │       └── .s3invsync.versions.json
        │           ├── 1/
        │           │   ├── .checksum
        │           │   ├── .checksum.old.7Bf7gvjmNgdtN8FUO4Qk90nyU8sy3QMS.5345d3a865378beead35816b57453e9d
        │           │   ├── .checksum.old.LSOSqtU6M0rZ.ZDmW5A4i27ICA2gTFqh.5345d3a865378beead35816b57453e9d
        │           │   ├── .checksum.old.YQpymw9KhVFiEkU.VxsOLB.6_JEYNnm3.5345d3a865378beead35816b57453e9d
        │           │   ├── .checksum.old.m10IbL0SiS9Bwl2y6J7O.7jTC8oBDMFx.5345d3a865378beead35816b57453e9d
        │           │   ├── .s3invsync.versions.json
        │           │   ├── 0/
        │           │   │   ├── .checksum
        │           │   │   ├── .checksum.old.2Tp4YVhtsI88wOvTDCuD5eAoLnozgTCB.604f787e8bc9c04aafc83fd277a3d101
        │           │   │   ├── .checksum.old.mzFRUYeYzvJYbabC96FMr91bOIxY6Hc..604f787e8bc9c04aafc83fd277a3d101
        │           │   │   ├── .checksum.old.niRMoBAoN50QPfA8IGQVlZcpl5aX7Aa5.604f787e8bc9c04aafc83fd277a3d101
        │           │   │   ├── .checksum.old.v96GwbwP36gg.y_fF0rWIqUBJ7LP6Vdy.604f787e8bc9c04aafc83fd277a3d101
        │           │   │   └── .s3invsync.versions.json
        │           │   ├── 1/
        │           │   │   ├── .checksum
        │           │   │   ├── .checksum.old.ItQg181onHVOuBenLCHQdCVBKpxZF9t2.cbe8c78be2d6acae94184ae7b450f8b9
        │           │   │   ├── .checksum.old.eBGR5XPSP_r_2zBF0mdJvL5U_D1f1._6.cbe8c78be2d6acae94184ae7b450f8b9
        │           │   │   ├── .checksum.old.gkBEk7odf13hk2A1C.dZLAaitUlBWjYT.cbe8c78be2d6acae94184ae7b450f8b9
        │           │   │   ├── .checksum.old.zgSa5BfduBNUyBGKubnvN4V9FIR0cRRN.cbe8c78be2d6acae94184ae7b450f8b9
        │           │   │   └── .s3invsync.versions.json
        │           │   ├── 2/
        │           │   │   ├── .checksum
        │           │   │   ├── .checksum.old.B9FMUf7mW.yhvzwupJ53YSxxV02kl4ZH.49974c572b0906ead73c29328eff367a
        │           │   │   ├── .checksum.old.KR0mDhO3tQcyZbSX2TWaNWYAHKUsl_8h.49974c572b0906ead73c29328eff367a
        │           │   │   ├── .checksum.old.Xqjr8eJqEMoqtV2UWwf1jnqeBRbKU59K.49974c572b0906ead73c29328eff367a
        │           │   │   ├── .checksum.old.hRyLoGOB3sFDczT5e26aWxJJjwGhoI7X.49974c572b0906ead73c29328eff367a
        │           │   │   └── .s3invsync.versions.json
        │           │   └── 3/
        │           │       ├── .checksum
        │           │       ├── .checksum.old.LERpq2tSEKGOmH2ERzoz3AXWu2ZdwgQU.ad5f04c479a411b0f23973c15701c4f3
        │           │       ├── .checksum.old.XRHyscUJUVjZyYlBhK2w0KpREt12WGrB.ad5f04c479a411b0f23973c15701c4f3
        │           │       ├── .checksum.old.YRKRm0HeLXABVHkdLzQ93bDf1QSSpQKP.ad5f04c479a411b0f23973c15701c4f3
        │           │       ├── .checksum.old.YZyR0weAVGDd3ec7Bw5JDWV60M31_0Xd.ad5f04c479a411b0f23973c15701c4f3
        │           │       └── .s3invsync.versions.json
        │           ├── 2/
        │           │   ├── .checksum
        │           │   ├── .checksum.old.1mjW3jRHvqVMpvsrV8vztZY8.E7e5vSi.1e8d7ac4628886e49a5ad493b42f4401
        │           │   ├── .checksum.old.DBi8uXyHF4HZt59t4Slkiu7fPJ8mjpBk.1e8d7ac4628886e49a5ad493b42f4401
        │           │   ├── .checksum.old.Vc2Q2RZpfKJDgHkjTewTC9NiyAF8mUGe.1e8d7ac4628886e49a5ad493b42f4401
        │           │   ├── .checksum.old.WkFEY_5r_V5OsNOEGBuEkzuYuInhGg_x.1e8d7ac4628886e49a5ad493b42f4401
        │           │   ├── .s3invsync.versions.json
        │           │   ├── 0/
        │           │   │   ├── .checksum
        │           │   │   ├── .checksum.old.RC_Lj4KVAYu9tHGDt2cl1cM.ybEaxswn.08bc49eef0f10f72afbaa11706e6ad4b
        │           │   │   ├── .checksum.old.ciMJemBRB2WExwpfYFmvkPo49vS21rWL.08bc49eef0f10f72afbaa11706e6ad4b
        │           │   │   ├── .checksum.old.gc2gn5DfhwS1uCHaHRikIfcqZHpoTG2w.08bc49eef0f10f72afbaa11706e6ad4b
        │           │   │   ├── .checksum.old.jcP_4YKDZvG5ti8jDCFP7h_fLdCyL4_K.08bc49eef0f10f72afbaa11706e6ad4b
        │           │   │   └── .s3invsync.versions.json
        │           │   ├── 1/
        │           │   │   ├── .checksum
        │           │   │   ├── .checksum.old.8_svUjfvYwgPEoLurmcH.ijR1WIyHGvT.9e9918b0b74d26af31244df42dfcb896
        │           │   │   ├── .checksum.old.C7lbx5I2gE7VfqFQ8oOUNFRSaCVowclA.9e9918b0b74d26af31244df42dfcb896
        │           │   │   ├── .checksum.old.FpPaCFkSc4YjgzFq20WNqU3qj7So25R8.9e9918b0b74d26af31244df42dfcb896
        │           │   │   ├── .checksum.old.y4lm0yAgDhAs6pxsD449eUL7T1mwwN9L.9e9918b0b74d26af31244df42dfcb896
        │           │   │   └── .s3invsync.versions.json
        │           │   ├── 2/
        │           │   │   ├── .checksum
        │           │   │   ├── .checksum.old.VEv2eU5fXGzaSCplYzU9EOQ_.9sCvVDG.8d8e054dd000303b458f42945ed4f6f0
        │           │   │   ├── .checksum.old.WP.2HKwRIEs1_ioYrIvTxMFEpCuHWPN5.8d8e054dd000303b458f42945ed4f6f0
        │           │   │   ├── .checksum.old.qK_QHzqKtCngDuhZFDvnLoz9EVak9jdd.8d8e054dd000303b458f42945ed4f6f0
        │           │   │   ├── .checksum.old.saV8VttLhYns5cm_Qq5er3FeTFPX5p8r.8d8e054dd000303b458f42945ed4f6f0
        │           │   │   └── .s3invsync.versions.json
        │           │   └── 3/
        │           │       ├── .checksum
        │           │       ├── .checksum.old.78NwH2yMgJY7BoMbn2q3hED.y7B1RQBW.0fab38ffccee060e924dd5420c355ddb
        │           │       ├── .checksum.old.82j82QvIfLXZ3MCocc4iBy_8IGV.Ulic.0fab38ffccee060e924dd5420c355ddb
        │           │       ├── .checksum.old.VABSQhgVcImiUqnexzXDOzH26._xqLBw.0fab38ffccee060e924dd5420c355ddb
        │           │       ├── .checksum.old.jhla51pwWt_4.CafsxUs.ltMGkh.S48a.0fab38ffccee060e924dd5420c355ddb
        │           │       └── .s3invsync.versions.json
        │           └── 3/
        │               ├── .checksum
        │               ├── .checksum.old..zBl9Igzhz8NCh.l7iIg0GCgp8ZcTFSN.b2b76eda321e6473a3e299bb8d99cd22
        │               ├── .checksum.old.BYEIlbCSdusv5It9rsoOFXn_THS__ZLF.34c0e22c4c73d2e0aee37dc0924550be
        │               ├── .checksum.old.BdoY24oGZeAl8gY02d9uzraJmzu1wJez.b2b76eda321e6473a3e299bb8d99cd22
        │               ├── .checksum.old.FpUH.LR9ZHc2HUjgWoi3XOEl5mdqfAI3.b2b76eda321e6473a3e299bb8d99cd22
        │               ├── .checksum.old._geTcSMWUxHBnnb5I6cOagSzh1E1Rr4Z.34c0e22c4c73d2e0aee37dc0924550be
        │               ├── .checksum.old.eKZ4.P8.1KMmfJq6qrzswvj7itfBFc_2.b2b76eda321e6473a3e299bb8d99cd22
        │               ├── .checksum.old.m2cqnKsKyfeB8jyQ7Yc47dfBVopdW6rA.34c0e22c4c73d2e0aee37dc0924550be
        │               ├── .checksum.old.rmuL4T451CM.L3X5pczi17fDnS7ZINRM.34c0e22c4c73d2e0aee37dc0924550be
        │               ├── .checksum.old.wal0IQ6ZdK6NVCctG0HtEGQie1hdz6LO.b2b76eda321e6473a3e299bb8d99cd22
        │               ├── .s3invsync.versions.json
        │               ├── 0/
        │               │   ├── .checksum
        │               │   ├── .checksum.old.9zXak.BYb3pz6xDApsYnWcW6kQTQuflW.749f5ef7a69500309eac52bab9b39173
        │               │   ├── .checksum.old.DaO5L7lcdCl1l5KWD7SwOj7sCQU38miz.749f5ef7a69500309eac52bab9b39173
        │               │   ├── .checksum.old.E1n.HJv4ch_Xl.kKs83NTdN_l5XNMm5H.1977bd3d2786e26cf0fb94bbe7477e9c
        │               │   ├── .checksum.old.SPdV4Olm9H9gXQc4iqA82Q.PdvQucfU..1977bd3d2786e26cf0fb94bbe7477e9c
        │               │   ├── .checksum.old.a7v7MUhUZlIrrc7nCQLF1dtLee5BJgKV.749f5ef7a69500309eac52bab9b39173
        │               │   ├── .checksum.old.keASeyl0O580oWK.sQ7mNR7mHMLYGu3s.749f5ef7a69500309eac52bab9b39173
        │               │   ├── .checksum.old.ttIPIb7yVLUhXvh0_x6ak3ioOfp9j7hY.1977bd3d2786e26cf0fb94bbe7477e9c
        │               │   ├── .checksum.old.yVJdcpQXDf6p_kwal02E9Ez92cpEkvsu.1977bd3d2786e26cf0fb94bbe7477e9c
        │               │   ├── .checksum.old.zta8oDaIMhPevtjV2vPQ1Z421Jby48dW.1977bd3d2786e26cf0fb94bbe7477e9c
        │               │   └── .s3invsync.versions.json
        │               ├── 1/
        │               │   ├── .checksum
        │               │   ├── .checksum.old.MfyI0MrXm2FnT4YRYxo6IxkbG9E9JDUj.a3ff7c79f534c129fd6a5aba4ace632c
        │               │   ├── .checksum.old.TDsv1jRoYffjpkxLPQS8rTJZ5LxjsCg4.a3ff7c79f534c129fd6a5aba4ace632c
        │               │   ├── .checksum.old.Zc8y1rlSuA481A1XaTw9EgZ1oyCVwzJU.a3ff7c79f534c129fd6a5aba4ace632c
        │               │   ├── .checksum.old.m05TxzCV7eeNxD.eVZBJdwEqYvox16LU.a3ff7c79f534c129fd6a5aba4ace632c
        │               │   └── .s3invsync.versions.json
        │               ├── 2/
        │               │   ├── .checksum
        │               │   ├── .checksum.old.PYKvAkREm7a5hebc1OHn7MyOFLU6Kh9j.7c5d9367276abd345bd1d4522a31aad9
        │               │   ├── .checksum.old.VSa50gfkNOxma8CXhhPO9dtS5rqqX9LO.7c5d9367276abd345bd1d4522a31aad9
        │               │   ├── .checksum.old.WDeN3WX6QTomFgv8vrWiCj8jBQmWDPNC.7c5d9367276abd345bd1d4522a31aad9
        │               │   ├── .checksum.old.ZFlZ8B2DkTqqXoArgxWMm7eE.kMU_WRV.7c5d9367276abd345bd1d4522a31aad9
        │               │   └── .s3invsync.versions.json
        │               └── 3/
        │                   ├── .checksum
        │                   ├── .checksum.old.1fC3.9OBLShOeh7sO_aRmo_j8HxqdoG5.4db4fb0d2a293f3afc107ae99d6dc0fa
        │                   ├── .checksum.old.7F9NK0lKPPQksijsbdwx3QfUjj2NBayS.4db4fb0d2a293f3afc107ae99d6dc0fa
        │                   ├── .checksum.old.CPCuRmrBssmTSiCuTBKxftkD5p9u8Kik.4db4fb0d2a293f3afc107ae99d6dc0fa
        │                   ├── .checksum.old.tcxR3qxKZTuy9tb51N8VU0AEsx7Y4_uP.4db4fb0d2a293f3afc107ae99d6dc0fa
        │                   └── .s3invsync.versions.json
        ├── 3/
        │   ├── .checksum
        │   ├── .checksum.old.BALcoKuuwnYLghOaoOrSr0CWsJbo9RVg.0a9dd9d5505822459d5c727d3c3dfa8e
        │   ├── .checksum.old.JU.uZHxDJdcbi8aKMbUQSW2dFCVQKgif.0a9dd9d5505822459d5c727d3c3dfa8e
        │   ├── .checksum.old.gzzT0UiCxIxbopf6AGmbYWe5PjSlabKi.0a9dd9d5505822459d5c727d3c3dfa8e
        │   ├── .checksum.old.zqNJYDr_dw_MQE8j6DrFkNw8BMvybO9I.0a9dd9d5505822459d5c727d3c3dfa8e
        │   ├── .s3invsync.versions.json
        │   └── 0/
        │       ├── .checksum
        │       ├── .checksum.old..8wTanAjUbUp7abCFIWjeRCeAAfM0Dqy.8af5fdf7b6afc6f8eea6d30b037edc8a
        │       ├── .checksum.old.6yaHGJ7VENo1ikxb0k6IoQ3iapI6e2EE.8af5fdf7b6afc6f8eea6d30b037edc8a
        │       ├── .checksum.old.LQNRjSGiQgcOlenlfD_JqKNS1aSLSUmk.8af5fdf7b6afc6f8eea6d30b037edc8a
        │       ├── .checksum.old.zkMqAkd4EGiOVGvwcougt2Lr8OqInofd.8af5fdf7b6afc6f8eea6d30b037edc8a
        │       ├── .s3invsync.versions.json
        │       └── 0/
        │           ├── .checksum
        │           ├── .checksum.old..oewjfKKMMWQ7UbsDRHD.wZvn61dyrhU.a18e4c3b50706fd7bde40e06845030cd
        │           ├── .checksum.old.7aRjEPVzHf52TsxcUh0uYnWAeksSulho.a18e4c3b50706fd7bde40e06845030cd
        │           ├── .checksum.old.MgpMSiapH7G563fylDWOlLH7wr9Gafyh.a18e4c3b50706fd7bde40e06845030cd
        │           ├── .checksum.old.NFolGcCMoF8HoXav6vJaCaV7C.nV1C9L.a18e4c3b50706fd7bde40e06845030cd
        │           ├── .s3invsync.versions.json
        │           ├── 0/
        │           │   ├── .checksum
        │           │   ├── .checksum.old.3yNuz4X5PmBXLf7UpwZjk9HRVBdryG2k.6ef2e0837f45aec661d639f229eb1db8
        │           │   ├── .checksum.old.S7LZDBEX21ulWpBWvjtvnCa_7twhaTQb.6ef2e0837f45aec661d639f229eb1db8
        │           │   ├── .checksum.old._Br4knor7vzHySUCootlWaHVEKmbijJJ.6ef2e0837f45aec661d639f229eb1db8
        │           │   ├── .checksum.old.x.FOZAL8zbT54l4hWtv8L8_jqZ1nqNVh.6ef2e0837f45aec661d639f229eb1db8
        │           │   ├── .s3invsync.versions.json
        │           │   ├── 0/
        │           │   │   ├── .checksum
        │           │   │   ├── .checksum.old.6dz6lbG3bm2EnEaR4piVKTs2bHoBQ5g7.719885911a6c9b801248044d0b87222e
        │           │   │   ├── .checksum.old.7z0QlMClZB_U5c3GhR8phFLC9m_bRJwY.719885911a6c9b801248044d0b87222e
        │           │   │   ├── .checksum.old.EqOGlcluqcZEv4rcXX5h2H1dib93HVgN.719885911a6c9b801248044d0b87222e
        │           │   │   ├── .checksum.old.cvCbqHty3RAH3QXV9QOJa1yIhleMLn0q.719885911a6c9b801248044d0b87222e
        │           │   │   └── .s3invsync.versions.json
        │           │   └── 1/
        │           │       ├── .checksum
        │           │       ├── .checksum.old.0ahsXHVrTPNzJ_mlH6.eSPa2.53OATlM.d0a28cc97fb24f9ac98950fbcd911070
        │           │       ├── .checksum.old.8.BhTwaBzrilqliilL7PqIdBSZZ8HLOf.d0a28cc97fb24f9ac98950fbcd911070
        │           │       ├── .checksum.old.GzLp_H_UJTLOTGcUCm09wWrXG1kzPEOT.d0a28cc97fb24f9ac98950fbcd911070
        │           │       ├── .checksum.old.tTwwTe.j3hlGMvH7QzH48yyPCIO7v0i_.d0a28cc97fb24f9ac98950fbcd911070
        │           │       └── .s3invsync.versions.json
        │           └── 1/
        │               ├── .checksum
        │               ├── .checksum.old.6gXUXhgtvkzmeaGERF34ogFA_axhYicy.9883855d69de7ecf566ccbf0529016e3
        │               ├── .checksum.old.AF_W3G6lxwo8mkAkmLpRcsYWNYLd6_LE.9883855d69de7ecf566ccbf0529016e3
        │               ├── .checksum.old.gyOE2WFs1S4.7rgF05A5yfx5bdl7OHay.9883855d69de7ecf566ccbf0529016e3
        │               ├── .checksum.old.zvFvin2rhnif76ObShkrxtnW8lSKgseR.9883855d69de7ecf566ccbf0529016e3
        │               ├── .s3invsync.versions.json
        │               ├── 0/
        │               │   ├── .checksum
        │               │   ├── .checksum.old.GLUBnW0I8afW1TJpCCAmbCYo1.dz7iaE.96b6293066c1d818d83e5f7add0f2162
        │               │   ├── .checksum.old.JLbx_vji67Xa8sMIZlLeXLUaTWtLXURs.96b6293066c1d818d83e5f7add0f2162
        │               │   ├── .checksum.old.UiXw1TA4GtYYbCFBTewdGlJiJoI4GBnE.96b6293066c1d818d83e5f7add0f2162
        │               │   ├── .checksum.old.na3uuRnlwBGXNFlb79SqaGT2pSVO7fhO.96b6293066c1d818d83e5f7add0f2162
        │               │   └── .s3invsync.versions.json
        │               └── 1/
        │                   ├── .checksum
        │                   ├── .checksum.old..Xe3kU991qBcdiPG7_Vl5IvDCaLe5fog.8cbeea47185754dc8e0b96535b0c04b1
        │                   ├── .checksum.old.P1hu4DOUI3d8gSyP4X65WR0TET470Tnu.8cbeea47185754dc8e0b96535b0c04b1
        │                   ├── .checksum.old.Y1HDbMcHa_Ef5lO55ef23TG3voPOV8W2.8cbeea47185754dc8e0b96535b0c04b1
        │                   ├── .checksum.old.fgxj3G9REIjt2x0JdI2tmlVqJw31TG1I.8cbeea47185754dc8e0b96535b0c04b1
        │                   └── .s3invsync.versions.json
        ├── 4/
        │   ├── .checksum
        │   ├── .checksum.old.Mcud0uSEcR2lv9Bn8NvVizjIx2pZorN4.af1086e3165ccd74778c319dd8af92e9
        │   ├── .checksum.old._7hXNKf_iUsp.2bwAVGIM7VxAp811nlh.af1086e3165ccd74778c319dd8af92e9
        │   ├── .checksum.old.lUrcv6XxQSqSrQGd5wT8VkFEWrAxcMZ3.af1086e3165ccd74778c319dd8af92e9
        │   ├── .checksum.old.rtsn_0pJhVMOTd3gDlLLCPJFu2xq2Ju7.af1086e3165ccd74778c319dd8af92e9
        │   ├── .s3invsync.versions.json
        │   └── 0/
        │       ├── .checksum
        │       ├── .checksum.old.0ItnYm2RefZTrzIlsq1TFXYZDRc3dHCz.f563fd78d6a8d571389809159f2037fb
        │       ├── .checksum.old.9rpr5Vr9Ew2TnJubkCE_Vl.Mb3fGIimb.f563fd78d6a8d571389809159f2037fb
        │       ├── .checksum.old.J6TF_hnzyNnd7klEocR5jm5SVeIU1Zf2.f563fd78d6a8d571389809159f2037fb
        │       ├── .checksum.old.wTdyPUbLxmv5xpeLQpqPP_gKu.1zTDeJ.f563fd78d6a8d571389809159f2037fb
        │       ├── .s3invsync.versions.json
        │       └── 0/
        │           ├── .checksum
        │           ├── .checksum.old.0C2WGtwsa.dhlij4lS4LKfu88tWfo0ah.eef02ad423e6dae897fad67406564546
        │           ├── .checksum.old.49DxB7SqOTlC6_Kvb7NozHHAisV_6NIJ.eef02ad423e6dae897fad67406564546
        │           ├── .checksum.old.4jRiS2nQbx8mH7JSU0iDCMdAZsk37z90.eef02ad423e6dae897fad67406564546
        │           ├── .checksum.old.a7mN8lwm2AjGaM.yhpBZPRC_7_0SXrwi.eef02ad423e6dae897fad67406564546
        │           ├── .s3invsync.versions.json
        │           └── 0/
        │               ├── .checksum
        │               ├── .checksum.old.BfvyljyR7rRJlj5AYZY0.OJQh_4Dx3FY.889f9a663e35ed0274241a89ce1accef
        │               ├── .checksum.old.Qmw2GtFt5UtKfp.OvmVBQASGQYrhWpmU.889f9a663e35ed0274241a89ce1accef
        │               ├── .checksum.old.RUYLpln1Bc9kZRvRTZE97gWe2tE_TQ3M.889f9a663e35ed0274241a89ce1accef
        │               ├── .checksum.old.w1wItgRq3pfhM1_IrrJdGsTilZFv.GoM.889f9a663e35ed0274241a89ce1accef
        │               ├── .s3invsync.versions.json
        │               └── 0/
        │                   ├── .checksum
        │                   ├── .checksum.old.0uHr7o1GziGtn1iFeY7R1KWVt24qKYiu.557b3c97ca90b476ba19379515183d24
        │                   ├── .checksum.old.mRxBxPx26B89CRkoE36Gq50DRpTx0.24.557b3c97ca90b476ba19379515183d24
        │                   ├── .checksum.old.ohacu69ncqshWFrXeCi.7ppioFrp4l73.557b3c97ca90b476ba19379515183d24
        │                   ├── .checksum.old.pzl8kZI63G2Oxp8i0frWd1nXVE5JxcEq.557b3c97ca90b476ba19379515183d24
        │                   └── .s3invsync.versions.json
        ├── 5/
        │   ├── .checksum
        │   ├── .checksum.old.29i3WSGsCyqpK85KW76Wyni_bd1f0fDz.f87edbe9ead086364600f64ad446c09a
        │   ├── .checksum.old.RXMu5z6__4pIIaM5UNLHkHakSCPbeNru.f87edbe9ead086364600f64ad446c09a
        │   ├── .checksum.old.kDNrHtIsWW5A9bNHiiy8zdPidlN1J9t..f87edbe9ead086364600f64ad446c09a
        │   ├── .checksum.old.wDmusZ8kU0CrSXtQ_SlN9lt1l2QPJjd9.f87edbe9ead086364600f64ad446c09a
        │   ├── .s3invsync.versions.json
        │   └── 0/
        │       ├── .checksum
        │       ├── .checksum.old.2Nl6_1K_Ex0srIRSjCeGdQXdR3pioyCW.274ec6abb94a26b8d261019babd5abfd
        │       ├── .checksum.old.LgHBfV_uEcE_5CCkGqsPI2TPkZWC2mQe.274ec6abb94a26b8d261019babd5abfd
        │       ├── .checksum.old.SMKgKfAiFk9vZvD2Vzat0ntEXFaYd5SB.274ec6abb94a26b8d261019babd5abfd
        │       ├── .checksum.old.r.EmTOKAOfDVhMvg3snh5T5NJZ68qmkF.274ec6abb94a26b8d261019babd5abfd
        │       ├── .s3invsync.versions.json
        │       └── 0/
        │           ├── .checksum
        │           ├── .checksum.old.QxYS8jrafq7naovVRplQ_555p2V4MqUS.5bfb28abaef98045e7f3020e5b8e4026
        │           ├── .checksum.old.gWXpsj4MFBZth15qD55gTo6HOJpk5SvS.5bfb28abaef98045e7f3020e5b8e4026
        │           ├── .checksum.old.m7R2QkYX9YlLqSJ0J35GVC9wuWLk9Bp2.5bfb28abaef98045e7f3020e5b8e4026
        │           ├── .checksum.old.n99ZnurNAxSEAZxj250Ux3rW7tkRSiT1.5bfb28abaef98045e7f3020e5b8e4026
        │           ├── .s3invsync.versions.json
        │           └── 0/
        │               ├── .checksum
        │               ├── .checksum.old.5lvlhOjt3SzvGEKlRLuHn4DXgdFp1ANj.82a06ee1a72b3f5ecccc1816b6928eb8
        │               ├── .checksum.old.7jQbH6hOLon0OmrCdPvqPV74IFDVDcLx.82a06ee1a72b3f5ecccc1816b6928eb8
        │               ├── .checksum.old.GbK5yI4a0SzAFlGpu_QsjS_k.2Kecc5Z.82a06ee1a72b3f5ecccc1816b6928eb8
        │               ├── .checksum.old.diAvTCJ.RAdCxjcFJTBjdWyfNTBS6fZc.82a06ee1a72b3f5ecccc1816b6928eb8
        │               ├── .s3invsync.versions.json
        │               └── 0/
        │                   ├── .checksum
        │                   ├── .checksum.old.BNdo8HdhctDCBdjcMV_3JxGbM7ETvmIY.eafe78c2c6995f5245333747ae27028e
        │                   ├── .checksum.old.EbZ5WfDu8.nhRuOYaetGR8p8N91XiYX1.eafe78c2c6995f5245333747ae27028e
        │                   ├── .checksum.old.TRfvU26qupScMF4oU549Ec8.8iuadm8C.eafe78c2c6995f5245333747ae27028e
        │                   ├── .checksum.old.Wq_icSuGy_W8PiQs6BABrw1129WdH1d2.eafe78c2c6995f5245333747ae27028e
        │                   └── .s3invsync.versions.json
        └── 6/
            ├── .checksum
            ├── .checksum.old.BPjV2N0RcRnaT9kIgT_6M4bCyxr9wYa3.50cac29dad460459ee19da7177b9fc59
            ├── .checksum.old.E0nuc0uAJoUKLHEj62VFZHLLBCq1Wyjg.50cac29dad460459ee19da7177b9fc59
            ├── .checksum.old.cZKAslQ4OSQvUvRgpx.0vvlmvLbufVRu.50cac29dad460459ee19da7177b9fc59
            ├── .checksum.old.z4yNYv.YW8JNku.1L8qUiuTeMCxr8vfW.50cac29dad460459ee19da7177b9fc59
            ├── .s3invsync.versions.json
            └── 0/
                ├── .checksum
                ├── .checksum.old.EREbwGjIRjScGOHZugY_udGY0SoBI8oV.fa462c2d86e5537a0355730cf2e0a3f8
                ├── .checksum.old.HbtraHXYoDEUOKm1oczEn6bXjwgTZVj1.fa462c2d86e5537a0355730cf2e0a3f8
                ├── .checksum.old.Pj903iqWnXOn1bVY6QG7V1KxlPYFVSsb.fa462c2d86e5537a0355730cf2e0a3f8
                ├── .checksum.old.mV25tLFGJr1mbLoUztE9JMfLWbiiy5Ct.fa462c2d86e5537a0355730cf2e0a3f8
                ├── .s3invsync.versions.json
                └── 0/
                    ├── .checksum
                    ├── .checksum.old.0o8P0ytnLJI8W0FCy3xknhTSx6ociJaH.db24000c29df2efd5e6eec19450f8cea
                    ├── .checksum.old.e_8DuinwJgimS_q1vZxdihW_hfKZMuEG.db24000c29df2efd5e6eec19450f8cea
                    ├── .checksum.old.etSJjxgrssXBpIF9Jhxpc.SA1raIDAli.db24000c29df2efd5e6eec19450f8cea
                    ├── .checksum.old.w5dKj2DwKzMq949gYJhUJrMOvQofkPG1.db24000c29df2efd5e6eec19450f8cea
                    ├── .s3invsync.versions.json
                    └── 0/
                        ├── .checksum
                        ├── .checksum.old.MQA7NLWIwPaucuRY7OVd.8LrMEZXzkRN.e82ccb64a84ef1d2d9218867962dd0e6
                        ├── .checksum.old.WAMkoyqOMGFrZ4D2ajFFmrsZVaf5M1pZ.e82ccb64a84ef1d2d9218867962dd0e6
                        ├── .checksum.old.dr_5W4F9RwAfCqJxM3LGUQrAVEz8rKMh.e82ccb64a84ef1d2d9218867962dd0e6
                        ├── .checksum.old.q88OPqW36ojnNviHfQ.iV72SIkdHCTTr.e82ccb64a84ef1d2d9218867962dd0e6
                        ├── .s3invsync.versions.json
                        └── 0/
                            ├── .checksum
                            ├── .checksum.old.AGMKrKbudkWWtx35_OVITrk8lbfhhx9B.9566d6818e294a8404b9df03b4685fb3
                            ├── .checksum.old.YaDpkNy.9Etvkr9Ow5a0B1DLxK7ygmOU.9566d6818e294a8404b9df03b4685fb3
                            ├── .checksum.old.hfgot5IjtT9xzljeBYpoTr9KgceSqd_y.9566d6818e294a8404b9df03b4685fb3
                            ├── .checksum.old.oNOQNBOwoIK.InTOjel.PBGil0TcO4wd.9566d6818e294a8404b9df03b4685fb3
                            └── .s3invsync.versions.json

(Some initial files are missing from the download because the inventory list started in the middle of the tree; if I had let the program run long enough, eventually it would have reached whatever CSV file listed the start of the tree.)

@jwodder jwodder merged commit e39ae10 into main Nov 25, 2024
9 checks passed
@jwodder jwodder deleted the gh-9 branch November 25, 2024 15:52
@yarikoptic
Copy link
Member

sweet! I will try running on typhon too now.

@yarikoptic
Copy link
Member

FTR, build done on my laptop with debian unstable and newer glibc is "not good" for typhon with older one

(3.13) (py3.13) yoh@typhon:/mnt/DATA/data/tmp/yoh/dandi$ ./s3invsync s3://dandiarchive/dandiarchive/dandiarchive/ dump 
./s3invsync: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by ./s3invsync)

@yarikoptic
Copy link
Member

found https://stackoverflow.com/questions/63724484/build-and-bind-against-older-libc-version -- will try later to build against musl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@jwodder jwodder

Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@jwodder @yarikoptic