[C++] Compatibility with newer aws sdk

[h-vetinari]

Describe the enhancement requested

Not sure whether to classify this as a bug or as an enhancement. It feels like a regression, but likely in AWS itself. In conda-forge, we don't build all patch versions of the SDK, because it gets released way too often to migrate ever time. Recently we went from 1.11.458 to 1.11.488, which ran into consistent errors on unix. I didn't look too closely at first (looked just like a flaky failure), but it reproduces 100% of the time. As such, I'm reverting the bump across our maintenance branches again.

Failures look roughly like this

[----------] 28 tests from TestS3FS
[ RUN      ] TestS3FS.GetFileInfoRoot
$SRC_DIR/cpp/src/arrow/filesystem/s3fs_test.cc:519: Failure
Failed
'PopulateTestBucket()' failed with IOError: AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding

[  FAILED  ] TestS3FS.GetFileInfoRoot (43 ms)
[ RUN      ] TestS3FS.GetFileInfoBucket
$SRC_DIR/cpp/src/arrow/filesystem/s3fs_test.cc:519: Failure
Failed
'PopulateTestBucket()' failed with IOError: AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding

[  FAILED  ] TestS3FS.GetFileInfoBucket (52 ms)

all due to malformed chunked encoding

This also shows up in the python tests (these are from arrow v15, which is the last version where we built the python bindings on the same feedstock; since the errors are uniform across versions though, I strongly suspect this to be the case also for main, as well as v19 and down).

=========================== short test summary info ============================
FAILED pyarrow/tests/test_fs.py::test_filesystem_is_functional_after_pickling[builtin_pickle-S3FileSystem] - OSError: When creating key 'a/' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_filesystem_is_functional_after_pickling[cloudpickle-S3FileSystem] - OSError: When creating key 'a/' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_dataset.py::test_write_dataset_s3 - OSError: When creating key 'dataset/' in bucket 'mybucket': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_create_dir[S3FileSystem] - OSError: When creating key 'test-directory/' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_copy_file[S3FileSystem] - OSError: When uploading part for key 'test-copy-source-file' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_move_file[S3FileSystem] - OSError: When uploading part for key 'test-move-source-file' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_delete_file[S3FileSystem] - OSError: When uploading part for key 'test-delete-target-file' in bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding
FAILED pyarrow/tests/test_fs.py::test_filesystem_from_uri_s3 - OSError: When creating key 'foo/' in bucket 'mybucket': AWS Error UNKNOWN (HTTP status 400) during PutObject operation: Unable to parse ExceptionName: BadRequest Message: malformed chunked encoding

Xref also #45195

Component(s)

C++, Python, Packaging

[pitrou]

The error message comes from Minio. So this is probably an incompatibility between Minio and the official AWS SDK... :(

[pitrou]

Ok, I can reproduce, and even with the latest Minio version is still occurs. So I'll probably have to run Wireshark on this...

[pitrou]

I have it down to this HTTP exchange:

• Request by AWS SDK:

PUT /bucket/emptydir/ HTTP/1.1
Host: 127.0.0.1:39491
Accept: */*
amz-sdk-invocation-id: B545DDA1-9E33-4987-B074-2B147AB8BD75
amz-sdk-request: attempt=1
authorization: AWS4-HMAC-SHA256 Credential=minio/20250120/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-type;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-trailer, Signature=5be6a95be231511f69fdc161843bbe83413ba52540e1607bc9de36ea42a827c5
content-encoding: aws-chunked
content-type: binary/octet-stream
transfer-encoding: chunked
user-agent: aws-sdk-cpp/1.11.488 ua/2.1 api/S3 os/Linux#5.15.0-130-generic lang/c++#C++11 md/aws-crt#0.29.9-dev+d5a3907e md/arch#x86_64 md/GCC#13.3.0
x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date: 20250120T094325Z
x-amz-decoded-content-length: 0
x-amz-trailer: x-amz-checksum-crc64nvme
Expect: 100-continue

• Response by Minio:

HTTP/1.1 400 Bad Request
Accept-Ranges: bytes
Content-Length: 331
Content-Type: application/xml
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: a4f5403139a65a8fd51b2aa0caa208291435f782459a282182de2ddf9eec98bf
X-Amz-Request-Id: 181C5D5BF688A7B0
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 11618
X-Ratelimit-Remaining: 11618
X-Xss-Protection: 1; mode=block
Date: Mon, 20 Jan 2025 09:43:25 GMT
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BadRequest</Code><Message>malformed chunked encoding</Message><Key>emptydir/</Key><BucketName>bucket</BucketName><Resource>/bucket/emptydir/</Resource><RequestId>181C5D5BF688A7B0</RequestId><HostId>a4f5403139a65a8fd51b2aa0caa208291435f782459a282182de2ddf9eec98bf</HostId></Error>

[pitrou]

Interesting, another PUT request before (to create a bucket) is successful.

• Request:

PUT /empty-bucket HTTP/1.1
Host: 127.0.0.1:39491
Accept: */*
amz-sdk-invocation-id: 54812C34-9549-4ECB-B54E-15B98759BCB9
amz-sdk-request: attempt=1
authorization: AWS4-HMAC-SHA256 Credential=minio/20250120/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=65a56d02027d24a6b29815814d051b012501bdb545bd32dc051f2f4c7b01ee7a
content-length: 0
content-type: application/xml
user-agent: aws-sdk-cpp/1.11.488 ua/2.1 api/S3 os/Linux#5.15.0-130-generic lang/c++#C++11 md/aws-crt#0.29.9-dev+d5a3907e md/arch#x86_64 md/GCC#13.3.0
x-amz-api-version: 2006-03-01
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20250120T094325Z

• Response:

HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 0
Location: /empty-bucket
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: a4f5403139a65a8fd51b2aa0caa208291435f782459a282182de2ddf9eec98bf
X-Amz-Request-Id: 181C5D5BF637BDF2
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 11618
X-Ratelimit-Remaining: 11618
X-Xss-Protection: 1; mode=block
Date: Mon, 20 Jan 2025 09:43:25 GMT

[xhochy]

Would it help your debugging if I build some of the versions in between on conda-forge?

[pitrou]

Probably not, thanks for the proposal.

[pitrou]

It appears there might be an easy workaround for this.

[pitrou]

Judging by https://httpwg.org/specs/rfc9112.html#chunked.encoding , this is really a bug in the AWS SDK. When using chunked encoding, there must be an explicit 0-sized chunk at the end, but it appears the AWS SDK client doesn't send one.

[h-vetinari]

Judging by https://httpwg.org/specs/rfc9112.html#chunked.encoding , this is really a bug in the AWS SDK. When using chunked encoding, there must be an explicit 0-sized chunk at the end, but it appears the AWS SDK client doesn't send one.

Thanks for digging into this so quickly! Could you open an issue in https://github.com/aws/aws-sdk-cpp? That would be useful for tracking/fixing this - if you want I can also do it, but I'd only be able to do a pale approximation, given that you understand the subject much better and did all the debugging.

[pitrou]

I'll open an issue, yes. I think I've found the culprit.

[pitrou]

Reported upstream at aws/aws-sdk-cpp#3259