Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download: more consistent and exhaustive logging, new DANDI_DEVEL_AGGRESSIVE_RETRY mode, respect (?) Retry-After #1509

Merged
merged 17 commits into from
Nov 15, 2024

Commits on Oct 24, 2024

  1. Log download directory path and also either we are entering __exit__ …

    …with or without exception etc
    yarikoptic committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    1286878 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    79f0f48 View commit details
    Browse the repository at this point in the history
  3. Add unit-testing for DownloadDirectory to ensure expected operation

    Also shortened the log line to not include traceback
    yarikoptic committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    d043359 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    15fde44 View commit details
    Browse the repository at this point in the history
  5. Move helper function to module level so it could be pickled

    not sure why was not failing for me locally but fails on CI
    yarikoptic committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    f436b0c View commit details
    Browse the repository at this point in the history
  6. Add handling of PermissionError on Windows for an existing directory

    That is my guess for what is happening in
    
        ________________________ test_DownloadDirectory_basic _________________________
        dandi\tests\test_download.py:1048: in test_DownloadDirectory_basic
            with DownloadDirectory(tmp_path, digests={}) as dl:
        dandi\download.py:889: in __exit__
            self.writefile.replace(self.filepath)
        C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\pathlib.py:1247: in replace
            self._accessor.replace(self, target)
        E   PermissionError: [WinError 5] Access is denied: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_DownloadDirectory_basic0.dandidownload\\file' -> 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_DownloadDirectory_basic0'
    yarikoptic committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    f7fadeb View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    028060e View commit details
    Browse the repository at this point in the history
  8. Make message more "factual"

    Very unlikely but it could be that directory already existed but without
    checksum file for some reason.
    yarikoptic committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    d3ce763 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    ddcab48 View commit details
    Browse the repository at this point in the history
  10. RF+ENH: also report requests and urllib3 versions, do not prefix with v

    no v prefix since is not providing any information on top; sorting for deterministic order
    yarikoptic committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    fb2e782 View commit details
    Browse the repository at this point in the history
  11. Also report python version

    yarikoptic committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    5a78454 View commit details
    Browse the repository at this point in the history
  12. Demand urllib3 to be no less than 2.0.0 in which enforce_content_leng…

    …th was set to default to True
    yarikoptic committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    a4db7bc View commit details
    Browse the repository at this point in the history
  13. ENH: respect/log separately Retry-After + support DANDI_DEVEL_AGGRESS…

    …IVE_RETRY mode
    
    This is all to address that odd case with 000026 where connection keeps interrupting.
    Unclear why so adding more specific cases handling and allowing for such an aggressive
    retrying where we would proceed as long as we are getting something (but sleep would also increase)
    yarikoptic committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    f52a363 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    a9226f0 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2024

  1. State minimal version of duecredit to be 0.6.0

    pip fails to deal with metadata of 0.5.0 due to missing __version__
    yarikoptic committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    ad069c8 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. Configuration menu
    Copy the full SHA
    08450ed View commit details
    Browse the repository at this point in the history
  2. fix(workaround,test): completely skip running test_DownloadDirectory_…

    …basic on windows
    
    Somehow that causes "indigestion" to pytest process later in its life cycle
    yarikoptic committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    7f97394 View commit details
    Browse the repository at this point in the history