Skip to content
This repository has been archived by the owner on Nov 19, 2024. It is now read-only.

Improve feedback when there's a validation error #44

Closed
dstansby opened this issue Apr 25, 2024 · 3 comments
Closed

Improve feedback when there's a validation error #44

dstansby opened this issue Apr 25, 2024 · 3 comments

Comments

@dstansby
Copy link
Member

I'm currently trying to upload a dataset, and if there's a validation error it's a bit tricky to understand what went wrong and how I can fix it. The output looks like this:

PATH                                                      SIZE        ERRORS     PROGRESS STATUS           MESSAGE                  
dandiset.yaml                                             1.1 kB                          skipped          should be edited online  
dataset_description.json                                  98 Bytes       2                ERROR            failed validation        
sub-s45/micr/sub-s45_sample-fullhemisphere_hipCT.json     65 Bytes       1                ERROR            failed validation        
sub-s45/micr/sub-s45_sample-fullhemisphere_hipCT.ome.zarr                2                ERROR            failed validation        
Summary:                                                  1.3 kB   3 with errors          1 skipped        1 should be edited online
                                                                                          3 ERROR          3 failed validation      
2024-04-25 13:44:09,332 [ WARNING] One or more assets failed validation.  Consult the logfile for details.
2024-04-25 13:44:09,861 [ WARNING] Upload failed, and you are not using the latest version of lincbrain-cli.  We suggest upgrading lincbrain-cli to v0.9.0 and trying again.
2024-04-25 13:44:09,864 [    INFO] Logs saved in /Users/dstansby/Library/Logs/lincbrain-cli/20240425124405Z-15907.log
Error: failed validation

If I then go into the log, I see messages like

024-04-25T13:44:09+0100 [WARNING ] dandi 15907:6207451136  Error 1: ValidationResult(id='BIDS.NON_BIDS_PATH_PLACEHOLDER', origin=ValidationOrigin(name='bidsschematools', version='0.7.2', bids_version='1.9.0-dev'), scope=<Scope.FILE: 'file'>, severity=<Severity.ERROR: 3>, asset_paths=None, within_asset_paths=None, dandiset_path=PosixPath('/Volumes/LaCie/linc_staging/000008'), dataset_path=PosixPath('/Volumes/LaCie/linc_staging/000008'), message='File does not match any pattern known to BIDS.', metadata=None, path=PosixPath('/Volumes/LaCie/linc_staging/000008/sub-s45/micr/sub-s45_sample-fullhemisphere_hipCT.json'), path_regex=None)
2024-04-25T13:44:09+0100 [ERROR   ] dandi 15907:6207451136 Error uploading /Volumes/LaCie/linc_staging/000008/sub-s45/micr/sub-s45_sample-fullhemisphere_hipCT.json:

These messages are hard to parse by eye, and even then the error is just "File does not match any pattern known to BIDS". It would be super useful if there was some hint (maybe at least a link to the BIDS spec?) on how the file pattern can be fixed.
@aaronkanzer
Copy link

@dstansby thanks for this -- was there any other output prior in your logs? Or just these two lines that you included in your ticket's description?

@dstansby
Copy link
Member Author

dstansby commented May 2, 2024

Here's an example full log where I got a validation error:

2024-05-02T14:40:51+0100 [INFO    ] dandi 23264:16516 lincbrain v0.5.0+108.g40a8c76b.dirty, hdmf v3.13.0, pynwb v2.6.0, h5py v3.11.0
2024-05-02T14:40:51+0100 [INFO    ] dandi 23264:16516 sys.argv = ['C:\\Users\\dstansby\\miniforge3\\envs\\linc\\Scripts\\lincbrain', 'upload']
2024-05-02T14:40:51+0100 [INFO    ] dandi 23264:16516 os.getcwd() = C:\Users\dstansby\data\linc\000005
2024-05-02T14:40:51+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 Starting new HTTPS connection (1): rig.mit.edu:443
2024-05-02T14:40:52+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 https://rig.mit.edu:443 "GET /et/projects/linc/linc-cli HTTP/1.1" 404 44
2024-05-02T14:40:52+0100 [DEBUG   ] dandi 23264:16516 Could not check linc/linc-cli for version updates: 404 Client Error: Not Found for url: https://rig.mit.edu/et/projects/linc/linc-cli
2024-05-02T14:40:52+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 Starting new HTTPS connection (1): api.lincbrain.org:443
2024-05-02T14:40:52+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 https://api.lincbrain.org:443 "GET /api/info/ HTTP/1.1" 200 371
2024-05-02T14:40:52+0100 [DEBUG   ] dandi 23264:16516 GET https://api.lincbrain.org/api/info/
2024-05-02T14:40:52+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 Starting new HTTPS connection (1): api.lincbrain.org:443
2024-05-02T14:40:53+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 https://api.lincbrain.org:443 "GET /api/info/ HTTP/1.1" 200 371
2024-05-02T14:40:53+0100 [DEBUG   ] dandi 23264:16516 Response: 200
2024-05-02T14:40:53+0100 [DEBUG   ] dandi 23264:16516 Using api key from LINCBRAIN_API_KEY environment variable
2024-05-02T14:40:53+0100 [DEBUG   ] dandi 23264:16516 GET https://api.lincbrain.org/api/auth/token
2024-05-02T14:40:53+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 https://api.lincbrain.org:443 "GET /api/auth/token HTTP/1.1" 301 0
2024-05-02T14:40:53+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 https://api.lincbrain.org:443 "GET /api/auth/token/ HTTP/1.1" 200 42
2024-05-02T14:40:53+0100 [DEBUG   ] dandi 23264:16516 Response: 200
2024-05-02T14:40:53+0100 [DEBUG   ] dandi 23264:16516 Found identifier DANDI:000005 in top level 'identifier'
2024-05-02T14:40:53+0100 [DEBUG   ] h5py._conv 23264:16516 Creating converter from 7 to 5
2024-05-02T14:40:53+0100 [DEBUG   ] h5py._conv 23264:16516 Creating converter from 5 to 7
2024-05-02T14:40:53+0100 [DEBUG   ] h5py._conv 23264:16516 Creating converter from 7 to 5
2024-05-02T14:40:53+0100 [DEBUG   ] h5py._conv 23264:16516 Creating converter from 5 to 7
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'zlib'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'gzip'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'bz2'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'lzma'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'blosc'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'zstd'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'lz4'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'astype'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'delta'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'quantize'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'fixedscaleoffset'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'packbits'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'categorize'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'pickle'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'base64'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'shuffle'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'bitround'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'crc32'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'adler32'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'jenkins_lookup3'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'json2'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'vlen-utf8'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'vlen-bytes'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'vlen-array'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'fletcher32'
2024-05-02T14:40:53+0100 [DEBUG   ] numcodecs 23264:16516 Registering codec 'n5_wrapper'
2024-05-02T14:40:54+0100 [INFO    ] dandi 23264:16516 Found 6 files to consider
2024-05-02T14:40:55+0100 [WARNING ] bids-schema 23264:17416 No BIDS reference root provided.
2024-05-02T14:40:55+0100 [INFO    ] bids-schema 23264:17416 No schema path specified, defaulting to the bundled schema, `C:\Users\dstansby\miniforge3\envs\linc\Lib\site-packages\bidsschematools\data\schema`.
2024-05-02T14:40:55+0100 [WARNING ] bids-schema 23264:17416 The `C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.ome.zarr/` file was not matched by any regex schema entry.
2024-05-02T14:40:56+0100 [WARNING ] dandi 23264:17416 'C:\\Users\\dstansby\\data\\linc\\000005\\dataset_description.json' had 1 validation errors preventing its upload:
2024-05-02T14:40:56+0100 [WARNING ] dandi 23264:17416  Error 1: ValidationResult(id='BIDS.NON_BIDS_PATH_PLACEHOLDER', origin=ValidationOrigin(name='bidsschematools', version='0.7.2', bids_version='1.9.0-dev'), scope=<Scope.FILE: 'file'>, severity=<Severity.ERROR: 3>, asset_paths=None, within_asset_paths=None, dandiset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), dataset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), message='File does not match any pattern known to BIDS.', metadata=None, path=WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.ome.zarr'), path_regex=None)
2024-05-02T14:40:56+0100 [ERROR   ] dandi 23264:17416 Error uploading C:\Users\dstansby\data\linc\000005\dataset_description.json:
Traceback (most recent call last):
  File "C:\Users\dstansby\miniforge3\envs\linc\Lib\site-packages\lincbrain\upload.py", line 280, in process_path
    raise UploadError("failed validation")
lincbrain.exceptions.UploadError: failed validation
2024-05-02T14:40:56+0100 [DEBUG   ] fscacher.cache 23264:29792 Calling memoized version of <function get_digest at 0x0000025AF771B7E0> for C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json
2024-05-02T14:40:56+0100 [DEBUG   ] fscacher.cache 23264:22488 Calling memoized version of <function get_digest at 0x0000025AF771B7E0> for C:\Users\dstansby\data\linc\000005\samples.tsv
2024-05-02T14:40:56+0100 [DEBUG   ] fscacher.cache 23264:30128 Calling memoized version of <function get_digest at 0x0000025AF771B7E0> for C:\Users\dstansby\data\linc\000005\participants.tsv
2024-05-02T14:40:56+0100 [DEBUG   ] fscacher.cache 23264:29792 Running original <function get_digest at 0x0000025AF771B7E0> on WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json')
2024-05-02T14:40:56+0100 [DEBUG   ] fscacher.cache 23264:29792 Calling memoized version of <function get_dandietag at 0x0000025AF77187C0> for C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:22488 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:30128 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/
2024-05-02T14:40:56+0100 [DEBUG   ] urllib3.connectionpool 23264:30128 Starting new HTTPS connection (2): api.lincbrain.org:443
2024-05-02T14:40:56+0100 [DEBUG   ] fscacher.cache 23264:29792 Running original <function get_dandietag at 0x0000025AF77187C0> on WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json')
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:29792 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/
2024-05-02T14:40:56+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 Starting new HTTPS connection (3): api.lincbrain.org:443
2024-05-02T14:40:56+0100 [DEBUG   ] urllib3.connectionpool 23264:22488 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/?path=samples.tsv HTTP/1.1" 200 274
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:22488 Response: 200
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:22488 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/d5c15a56-afd6-4b3b-a37b-b052849dbfc1/
2024-05-02T14:40:56+0100 [DEBUG   ] urllib3.connectionpool 23264:22488 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/d5c15a56-afd6-4b3b-a37b-b052849dbfc1/ HTTP/1.1" 200 1911
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:22488 Response: 200
2024-05-02T14:40:56+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/?path=rawdata%2Fsub-S45%2Fsub-S45_sample-fullhemisphere_PC.json HTTP/1.1" 200 52
2024-05-02T14:40:56+0100 [DEBUG   ] dandi 23264:29792 Response: 200
2024-05-02T14:40:57+0100 [DEBUG   ] urllib3.connectionpool 23264:30128 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/?path=participants.tsv HTTP/1.1" 200 278
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:30128 Response: 200
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:30128 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/aef8d0c9-0cd3-46d6-a2ac-3dfdee3fa06b/
2024-05-02T14:40:57+0100 [DEBUG   ] fscacher.cache 23264:29792 Calling memoized version of <function get_dandietag at 0x0000025AF77187C0> for C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:29792 Calculated dandi-etag of df515b2ad6855b779e78e4ec58fdd98f-1 for C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Beginning upload
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:29792 POST https://api.lincbrain.org/api/uploads/initialize/
2024-05-02T14:40:57+0100 [DEBUG   ] urllib3.connectionpool 23264:30128 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/aef8d0c9-0cd3-46d6-a2ac-3dfdee3fa06b/ HTTP/1.1" 200 1915
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:30128 Response: 200
2024-05-02T14:40:57+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 200 649
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:29792 Response: 200
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:29792 Uploading C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json in 1 parts
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:17160 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Uploading part 1/1 (104 bytes)
2024-05-02T14:40:57+0100 [DEBUG   ] dandi 23264:17160 PUT https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com/blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=content-length%3Bhost&X-Amz-Signature=c431ce73c38b9c8c2ffe2ebbfb08e06ee5b5274731c6b0b06bd56ec39efa48b9
2024-05-02T14:40:57+0100 [DEBUG   ] urllib3.connectionpool 23264:17160 Starting new HTTPS connection (1): linc-brain-mit-prod-us-east-2.s3.amazonaws.com:443
2024-05-02T14:40:58+0100 [DEBUG   ] urllib3.connectionpool 23264:17160 https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com:443 "PUT /blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=content-length%3Bhost&X-Amz-Signature=c431ce73c38b9c8c2ffe2ebbfb08e06ee5b5274731c6b0b06bd56ec39efa48b9 HTTP/1.1" 200 0
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:17160 Response: 200
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:17160 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Part upload finished ETag=6a057fca28c6d756dae00c6160579d31 Content-Length=0
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Completing upload
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 POST https://api.lincbrain.org/api/uploads/082cda3f-a994-4728-bf34-8108ab122bed/complete/
2024-05-02T14:40:58+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "POST /api/uploads/082cda3f-a994-4728-bf34-8108ab122bed/complete/ HTTP/1.1" 200 768
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 Response: 200
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Announcing completion to https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com/blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=3b46212653ca7ccabaeae1a69ddce210dde5e5b6f29e186495fad5dac065b4b1
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 POST https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com/blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=3b46212653ca7ccabaeae1a69ddce210dde5e5b6f29e186495fad5dac065b4b1
2024-05-02T14:40:58+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com:443 "POST /blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=3b46212653ca7ccabaeae1a69ddce210dde5e5b6f29e186495fad5dac065b4b1 HTTP/1.1" 200 None
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 Response: 200
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Upload completed. Response content: b'<?xml version="1.0" encoding="UTF-8"?>\n\n<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com/blobs%2F082%2Fcda%2F082cda3f-a994-4728-bf34-8108ab122bed</Location><Bucket>linc-brain-mit-prod-us-east-2</Bucket><Key>blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed</Key><ETag>"df515b2ad6855b779e78e4ec58fdd98f-1"</ETag></CompleteMultipartUploadResult>'
2024-05-02T14:40:58+0100 [DEBUG   ] charset_normalizer 23264:29792 Encoding detection: utf_8 is most likely the one.
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 POST https://api.lincbrain.org/api/uploads/082cda3f-a994-4728-bf34-8108ab122bed/validate/
2024-05-02T14:40:58+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "POST /api/uploads/082cda3f-a994-4728-bf34-8108ab122bed/validate/ HTTP/1.1" 200 119
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 Response: 200
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Assigning asset blob to dandiset & version
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 POST https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/
2024-05-02T14:40:58+0100 [DEBUG   ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "POST /api/dandisets/000005/versions/draft/assets/ HTTP/1.1" 200 2182
2024-05-02T14:40:58+0100 [DEBUG   ] dandi 23264:29792 Response: 200
2024-05-02T14:40:58+0100 [INFO    ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Asset successfully uploaded
2024-05-02T14:41:05+0100 [WARNING ] dandi 23264:1616 'C:\\Users\\dstansby\\data\\linc\\000005\\rawdata\\sub-S45\\sub-S45_sample-fullhemisphere_PC.ome.zarr' had 2 validation errors preventing its upload:
2024-05-02T14:41:05+0100 [WARNING ] dandi 23264:1616  Error 1: ValidationResult(id='BIDS.NON_BIDS_PATH_PLACEHOLDER', origin=ValidationOrigin(name='bidsschematools', version='0.7.2', bids_version='1.9.0-dev'), scope=<Scope.FILE: 'file'>, severity=<Severity.ERROR: 3>, asset_paths=None, within_asset_paths=None, dandiset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), dataset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), message='File does not match any pattern known to BIDS.', metadata=None, path=WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.ome.zarr'), path_regex=None)
2024-05-02T14:41:05+0100 [WARNING ] dandi 23264:1616  Error 2: ValidationResult(id='BIDS.NON_BIDS_PATH_PLACEHOLDER', origin=ValidationOrigin(name='bidsschematools', version='0.7.2', bids_version='1.9.0-dev'), scope=<Scope.FILE: 'file'>, severity=<Severity.ERROR: 3>, asset_paths=None, within_asset_paths=None, dandiset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), dataset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), message='File does not match any pattern known to BIDS.', metadata=None, path=WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.ome.zarr'), path_regex=None)
2024-05-02T14:41:05+0100 [ERROR   ] dandi 23264:1616 Error uploading C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.ome.zarr:
Traceback (most recent call last):
  File "C:\Users\dstansby\miniforge3\envs\linc\Lib\site-packages\lincbrain\upload.py", line 280, in process_path
    raise UploadError("failed validation")
lincbrain.exceptions.UploadError: failed validation
2024-05-02T14:41:05+0100 [WARNING ] dandi 23264:16516 One or more assets failed validation.  Consult the logfile for details.
2024-05-02T14:41:05+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 Starting new HTTPS connection (1): rig.mit.edu:443
2024-05-02T14:41:06+0100 [DEBUG   ] urllib3.connectionpool 23264:16516 https://rig.mit.edu:443 "GET /et/projects/lincbrain/linc-cli HTTP/1.1" 200 578
2024-05-02T14:41:06+0100 [WARNING ] dandi 23264:16516 Upload failed, and you are not using the latest version of lincbrain-cli.  We suggest upgrading lincbrain-cli to v0.9.0 and trying again.
2024-05-02T14:41:06+0100 [DEBUG   ] dandi 23264:16516 Caught exception failed validation
2024-05-02T14:41:06+0100 [INFO    ] dandi 23264:16516 Logs saved in C:\Users\dstansby\AppData\Local\lincbrain\lincbrain-cli\Logs\20240502134051Z-23264.log

@kabilar kabilar mentioned this issue Jul 24, 2024
Open
@kabilar
Copy link
Member

kabilar commented Jul 24, 2024

Thanks for the suggestion, @dstansby. I reposted your suggestion to the upstream DANDI Client repo (dandi#1469). Once the DANDI team provides an update, we will incorporate those changes into the LINC Client. I will go ahead and close this issue.

@kabilar kabilar closed this as completed Jul 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kabilar @dstansby @aaronkanzer