presigned urls issued by boto3 don't work

[evgeniiz321]

To get the url (an example from docs - https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html#presigned-urls):

self.s3_client.generate_presigned_url(
                    ClientMethod="get_object",
                    Params={"Bucket": bucket, "Key": file_name},
                    ExpiresIn=30,
                    HttpMethod="GET",
                )

This is the url that is returned:

https://localhost:55858/3e26a4ca-595d-4b4d-b592-424d9d739ccd/temp_file_3f2a06c4-77dd-4008-8968-8a384c6f19f0?AWSAccessKeyId=Ci5Z5p4Q56fzfgsmtUsXkMhys4FxM7izUQBmkWx7jUJx0DMygH7VmUxBVVCsAmAmfascvg4sG5kavjTaXG8qa2pok&Signature=l%2B1CRy9QiY%2Bl%2F2iQAQ70UxcLerU%3D&Expires=1732573443

But during GET I receive:

error	handler/util.go:34	call method	{"status": 403, "request_id": "766cd4cd-11c6-4535-ba97-78762d239613", "method": "GetObject", "bucket": "3e26a4ca-595d-4b4d-b592-424d9d739ccd", "object": "temp_file_3f2a06c4-77dd-4008-8968-8a384c6f19f0", "description": "could not find object", "error": "access denied: rpc error: code = Unknown desc = access to operation GET is denied by extended ACL check: DENY eACL rule"}

With URLs returned by authmate and aws s3 cli everything works as expected.

Can be reproduced by the test here - nspcc-dev/neofs-testcases#896

[smallhive]

Freshly installed env

$ pip freeze
boto3==1.35.72
botocore==1.35.72
certifi==2024.8.30
charset-normalizer==3.4.0
idna==3.10
jmespath==1.0.1
python-dateutil==2.9.0.post0
requests==2.32.3
s3transfer==0.10.4
six==1.16.0
urllib3==2.2.3

with script

import boto3
import requests

session = boto3.Session()
access_key_id = "DBhNx2FVFmLoAUzjJDMzWWV4ECCXygmKQfXPz5JqCAez0nkQ1oTtvWbxD9iuhDfwKPKEUvDUUQQdxuUhbdvrkmdU"
secret_access_key = "3296940caefe413b76688ab8067fcbd19ba08844cf9573659b9cac472ab8f24d"

s3_client = session.client(
    service_name="s3",
    aws_access_key_id=access_key_id,
    aws_secret_access_key=secret_access_key,
    endpoint_url=f"http://localhost:19080",
)

url = s3_client.generate_presigned_url(ClientMethod="get_object",
                                       Params={'Bucket': "heh1733115156", 'Key': "m1733115167"}, ExpiresIn=30,
                                       HttpMethod="GET", )
print(url)

resp = requests.get(url)
print(resp.text)

gives

http://localhost:19080/heh1733115156/m1733115167?AWSAccessKeyId=DBhNx2FVFmLoAUzjJDMzWWV4ECCXygmKQfXPz5JqCAez0nkQ1oTtvWbxD9iuhDfwKPKEUvDUUQQdxuUhbdvrkmdU&Signature=Lv%2BkXvmMv6KSNgWNoW4eZEvNU8E%3D&Expires=1733117902
aaazzzcccdddeee

aaazzzcccdddeee - is a valid object content

Could you please check something like this? Would It be an issue with secrets or maybe some misconfiguration?

[smallhive]

The mystery has been solved.The proper boto3 lib configuration is required.

The default example for boto3 generates the next URL:

https://localhost:60893/heh1733292366/fileToS32.txt?AWSAccessKeyId=HxEMaAtRRAdKHF8obnnBVQC5z1vpT8mDTKV2PuNwtfkB05ifRBi1w5qiD8sRYznjGDAtppPdFJmrRBiE5ouPX3Hgq&Signature=Os%2FyV5vbBXkm0ovY4W7P%2FW34scI%3D&Expires=1733298610

Meanwhile, the AWS CLI generates a bit different one:

$ aws s3 presign s3://heh1733292366/fileToS32.txt --expires-in 3600 --endpoint-url https://localhost:60893

https://localhost:60893/heh1733292366/fileToS32.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=HxEMaAtRRAdKHF8obnnBVQC5z1vpT8mDTKV2PuNwtfkB05ifRBi1w5qiD8sRYznjGDAtppPdFJmrRBiE5ouPX3Hgq%2F20241204%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241204T073613Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=a5f2adcf7c28f9fe5635fb809ef6403fac773fdc97e720727bdbcfac62a9d5b8

Much longer parameters, because it uses v4 signature. The boto3 lib can do the same, but It requires some configuration:

my_config = Config(
signature_version = 'v4',
)

import boto3
import requests
from botocore.config import Config

session = boto3.Session()
access_key_id = "HxEMaAtRRAdKHF8obnnBVQC5z1vpT8mDTKV2PuNwtfkB05ifRBi1w5qiD8sRYznjGDAtppPdFJmrRBiE5ouPX3Hgq"
secret_access_key = "dec72f51d988606b2a996d64db056399f2bccd975e5de9d99b113d9dc63fb192"

my_config = Config(
    signature_version = 'v4',
)

s3_client = session.client(
    service_name="s3",
    aws_access_key_id=access_key_id,
    aws_secret_access_key=secret_access_key,
    endpoint_url=f"https://localhost:60893",
    config=my_config
)

url = s3_client.generate_presigned_url(ClientMethod="get_object",
                                       Params={'Bucket': "heh1733292366", 'Key': "fileToS32.txt"}, ExpiresIn=30,
                                       HttpMethod="GET", )
print(url)

resp = requests.get(url)
print(resp.text)

With this option generated URL is fine:

https://localhost:60893/heh1733292366/fileToS32.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=HxEMaAtRRAdKHF8obnnBVQC5z1vpT8mDTKV2PuNwtfkB05ifRBi1w5qiD8sRYznjGDAtppPdFJmrRBiE5ouPX3Hgq%2F20241204%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241204T075424Z&X-Amz-Expires=30&X-Amz-SignedHeaders=host&X-Amz-Signature=f48f0287ae1cc5842dbc5413adb5a80f0dbd036ea7ccec725998e56fcdf36bb6

and perfectly works

[evgeniiz321]

All good, thank you.