Milvus failed to check blob bucket exist

[mahaveer08]

Environment

• Milvus version: v2.3.11
• Deployment mode(standalone or cluster): cluster

minio:
  enabled: false

externalS3:
  enabled: true
  host: "s3.us-east-2.amazonaws.com"
  port: "443"
  useSSL: true
  bucketName: "milvus-bucket"
  useIAM: false

Current Behavior

Having issue to connect milvus database as querynode shows below errors as I am unable establish a connection to the database:

[2024/03/08 19:58:03.292 +00:00] [WARN] [storage/minio_object_storage.go:124] ["failed to check blob bucket exist"] [bucket=milvus-bucket] [error="Access Denied."]
[2024/03/08 19:58:03.696 +00:00] [WARN] [storage/minio_object_storage.go:124] ["failed to check blob bucket exist"] [bucket=milvus-bucket] [error="Access Denied."]
[2024/03/08 19:58:04.501 +00:00] [WARN] [storage/minio_object_storage.go:124] ["failed to check blob bucket exist"] [bucket=milvus-bucket] [error="Access Denied."]
[2024/03/08 19:58:06.105 +00:00] [WARN] [storage/minio_object_storage.go:124] ["failed to check blob bucket exist"] [bucket=milvus-bucket] [error="Access Denied."]
[2024/03/08 19:58:06.105 +00:00] [ERROR] [storage/minio_object_storage.go:141] ["retry func failed"] ["retry time"=4] [error="Access Denied."] [stack="github.com/milvus-io/milvus/internal/storage.newMinioClient\n\t/go/src/github.com/milvus-io/milvus/internal/storage/minio_object_storage.go:141\ngithub.com/milvus-io/milvus/internal/storage.newMinioObjectStorageWithConfig\n\t/go/src/github.com/milvus-io/milvus/internal/storage/minio_object_storage.go:149\ngithub.com/milvus-io/milvus/internal/storage.NewRemoteChunkManager\n\t/go/src/github.com/milvus-io/milvus/internal/storage/remote_chunk_manager.go:74\ngithub.com/milvus-io/milvus/internal/storage.(*ChunkManagerFactory).newChunkManager\n\t/go/src/github.com/milvus-io/milvus/internal/storage/factory.go:54\ngithub.com/milvus-io/milvus/internal/storage.(*ChunkManagerFactory).NewPersistentStorageChunkManager\n\t/go/src/github.com/milvus-io/milvus/internal/storage/factory.go:61\ngithub.com/milvus-io/milvus/internal/util/dependency.(*DefaultFactory).NewPersistentStorageChunkManager\n\t/go/src/github.com/milvus-io/milvus/internal/util/dependency/factory.go:152\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:286\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:255\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:128\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:223\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:115"]

[bucket=milvus-bucket] [error="Get \"
[http://s3.us-east-2.amazonaws.com:443/milvus-bucket/?location=\":](http://s3.us-east-2.amazonaws.com:443/milvus-bucket/?location=\%22:)
net/http: HTTP/1.x transport connection broken: malformed HTTP response \"\\x15\\x03\\x01\\x00\\x02\\x01\\x

Pod Status

milvus-demo-datacoord-1                     0/1     Running     3 (37s ago)   5m37s
milvus-demo-datacoord-2                     1/1     Running     0             5m36s
milvus-demo-datanode-1                      0/1     Running     2 (14s ago)   5m36s
milvus-demo-etcd-0                              1/1     Running     0             5m35s
milvus-demo-etcd-1                              1/1     Running     0             5m35s
milvus-demo-etcd-2                              1/1     Running     0             5m35s
milvus-demo-indexcoord-1                   1/1     Running     0             5m37s
milvus-demo-indexcoord-2                   1/1     Running     0             5m36s
milvus-demo-indexnode-1                    1/1     Running     0             5m36s
milvus-demo-proxy-1                            0/1     Running     1 (66s ago)   5m36s
milvus-demo-proxy-1                            0/1     Running     1 (66s ago)   5m37s
milvus-demo-pulsar-bookie-0               1/1     Running     0             5m36s
milvus-demo-pulsar-bookie-1               1/1     Running     0             5m35s
milvus-demo-pulsar-bookie-2               1/1     Running     0             5m35s
milvus-demo-pulsar-bookie-init-1         0/1     Completed   0             5m35s
milvus-demo-pulsar-broker-0                1/1     Running     0             5m36s
milvus-demo-pulsar-proxy-0                 1/1     Running     0             5m36s
milvus-demo-pulsar-pulsar-init-1          0/1     Completed   0             5m35s
milvus-demo-pulsar-recovery-0            1/1     Running     0             5m36s
milvus-demo-pulsar-zookeeper-0         1/1     Running     0             5m36s
milvus-demo-pulsar-zookeeper-1         1/1     Running     0             4m57s
milvus-demo-pulsar-zookeeper-2         1/1     Running     0             4m18s
milvus-demo-querycoord-1                   0/1     Running     1 (65s ago)   5m37s
milvus-demo-querycoord-2                   0/1     Running     1 (66s ago)   5m36s
milvus-demo-querynode-1                    0/1     Error       3 (74s ago)   5m36s
milvus-demo-rootcoord-1                     1/1     Running     0             5m37s
milvus-demo-rootcoord-2                     1/1     Running     0             5m36s

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

it seems to be a access denied, did you by chance config your AK/SK?

[xiaofan-luan]

it is configured by minio.accessKeyID and minio.secretAccessKey

[xiaofan-luan]

true.

transport connection broken: malformed HTTP response might means some protocol incompatible.
Are you enabling TLS on S3 side?

[mahaveer08]

Yes this is my bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EnforceTLSv12orHigher",
            "Effect": "Deny",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::milvus-bucket/*",
                "arn:aws:s3:::milvus-bucket"
            ],
            "Condition": {
                "NumericLessThan": {
                    "s3:TlsVersion": "1.2"
                }
            }
        }
    ]
}

Not sure what I am missing tried everything but no luck yet.

[xiaofan-luan]

"Condition": {
"NumericLessThan": {
"s3:TlsVersion": "1.2"
}
}

That looks weird, see you set tls enabled to true, but
http://s3.us-east-2.amazonaws.com:443/milvus-bucket/?location=":
net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x01\x

you are still using http endpoint rather than https

[mahaveer08]

So what should I need to change and where to change?

[sharbuz-truststamp]

I have the same issue when trying to backup from EBS hosted Minio to S3.

root@minio-migration:/# vim configs/backup.yaml root@minio-migration:/# ./milvus-backup check 0.4.26 (Built on 2024-10-30T09:28:19Z from Git SHA dd4515017a2d2f7c9493070fe321156a6c9126d4) config:backup.yaml [2024/11/21 10:56:34.491 +00:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=] [2024/11/21 10:56:34.491 +00:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log] [2024/11/21 10:56:34.501 +00:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=milvus-bucket] [root=file] [2024/11/21 10:56:54.521 +00:00] [WARN] [storage/minio_chunk_manager.go:115] ["failed to check blob bucket exist"] [bucket=milvus-xxxxxxxx-local] [error="Get \"http://s3.us-west-2.amazonaws.com:443/milvus-xxxxxxxx-local/?location=\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \"\\x15\\x03\\x01\\x00\\x02\\x01\\x00\""] [2024/11/21 10:57:14.729 +00:00] [WARN] [storage/minio_chunk_manager.go:115] ["failed to check blob bucket exist"] [bucket=milvus-xxxxxxxx-local] [error="Get \"http://s3.us-west-2.amazonaws.com:443/milvus-xxxxxxxx-local/?location=\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \"\\x15\\x03\\x01\\x00\\x02\\x01\\x00\""] ^C root@minio-migration:/#

but in the same time backup between S3 buckets works without error, the problem is in useSSL, that set HTTPS for both endpoints

[xiaofan-luan]

is there a log like this?
"init AwsChunkManager with "
"parameter[endpoint={}][bucket_name={}][root_path={}][use_secure={}]",

[xiaofan-luan]

check the value of use_secure

[mahaveer08]

From which pod need to get that?

[xiaofan-luan]

Should be on querynode

[xiaofan-luan]

you can use export log.sh to collect logs

[xiaofan-luan]

AwsChunkManager::AwsChunkManager(const StorageConfig& storage_config) {
default_bucket_name_ = storage_config.bucket_name;
remote_root_path_ = storage_config.root_path;

InitSDKAPIDefault(storage_config.log_level);

Aws::Client::ClientConfiguration config = generateConfig(storage_config);
if (storage_config.useIAM) {
    auto provider =
        std::make_shared<Aws::Auth::DefaultAWSCredentialsProviderChain>();
    auto aws_credentials = provider->GetAWSCredentials();
    AssertInfo(!aws_credentials.GetAWSAccessKeyId().empty(),
               "if use iam, access key id should not be empty");
    AssertInfo(!aws_credentials.GetAWSSecretKey().empty(),
               "if use iam, secret key should not be empty");
    AssertInfo(!aws_credentials.GetSessionToken().empty(),
               "if use iam, token should not be empty");

    client_ = std::make_shared<Aws::S3::S3Client>(
        provider,
        config,
        Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never,
        storage_config.useVirtualHost);
} else {
    BuildAccessKeyClient(storage_config, config);
}

PreCheck(storage_config);

LOG_INFO(
    "init AwsChunkManager with "
    "parameter[endpoint={}][bucket_name={}][root_path={}][use_secure={}]",
    storage_config.address,
    storage_config.bucket_name,
    storage_config.root_path,
    storage_config.useSSL);

}

[xiaofan-luan]

Aws::Client::ClientConfiguration
generateConfig(const StorageConfig& storage_config) {
// The ClientConfiguration default constructor will take a long time.
// For more details, please refer to aws/aws-sdk-cpp#1440
static Aws::Client::ClientConfiguration g_config;
Aws::Client::ClientConfiguration config = g_config;
config.endpointOverride = ConvertToAwsString(storage_config.address);

if (storage_config.useSSL) {
    config.scheme = Aws::Http::Scheme::HTTPS;
    config.verifySSL = true;
} else {
    config.scheme = Aws::Http::Scheme::HTTP;
    config.verifySSL = false;
}

if (!storage_config.region.empty()) {
    config.region = ConvertToAwsString(storage_config.region);
}

config.requestTimeoutMs = storage_config.requestTimeoutMs == 0
                              ? DEFAULT_CHUNK_MANAGER_REQUEST_TIMEOUT_MS
                              : storage_config.requestTimeoutMs;

return config;

}

[xiaofan-luan]

if usessl is set, then you should definitely use Aws::Http::Scheme::HTTPS rather than Aws::Http::Scheme::HTT

[xiaofan-luan]

@LoveEachDay anything you can run to check the config itself is correct?

[mahaveer08]

query_node.log

[mahaveer08]

I think I am missing some simple thing but not sure what is that? For serviceAccount should we need to attach to eks cluster?

[xiaofan-luan]

I believe this is not a issue of Milvus itself but more like a HELM chart issue, even the golang minio chunk manager is inited as expected
Unfortunately I'm not an expert on it.
Will let someone take a look./

[xiaofan-luan]

@haorenfsa
any ideas on this?

[mahaveer08]

I am trying to deploy milvus on existing eks cluster and followed this document: https://milvus.io/docs/eks.md

[mahaveer08]

Any updates?

[haorenfsa]

Hi @mahaveer08! According to the query_node.log you provided. It shows you're using the bucket named milvus-bucket

Which I just checked, is located in ap-southeast-1. Are you sure you have the access to this bucket? or is this a mistake?
Could also provide the helm values you used to run the milvus which produced the query_node.log

[haorenfsa]

Before you reply, I assume you're using below command:

helm upgrade --install --set cluster.enabled=true --set dataNode.replicas=1 --set indexNode.replicas=1 --set queryNode.replicas=1 --set proxy.replicas=1 --set externalS3.enabled=true --set externalS3.host='s3.us-west-2.amazonaws.com' --set externalS3.port=80 --set externalS3.accessKey=<aws_AK> --set externalS3.secretKey=<aws_SK>--set externalS3.bucketName=milvus-bucket --set minio.enabled=False --set service.type=LoadBalancer milvus milvus/milvus

First make sure the externalS3.bucketName is set to the bucket that owned by your AWS account.

Then check whether your externalS3.accessKey & externalS3.secretKey has the access to the bucket externalS3.bucketName. The Access Denied error suggest a No answer.

Then make sure the host externalS3.host is of the same region of your bucket. You can try the host name http://milvus-bucket.s3.us-east-2.amazonaws.com/ in your browser. Note to change milvus-bucket to your own bucket here. There should not be a PermanentRedirect error.

[mahaveer08]

I am able to resolve the bucket issue(It was the role assignment issue with service account) and all pods are up and running but when try to connect to milvus db using loadbalancer external IP I get below error:

E0311 13:58:54.277000000 31616 src/core/tsi/ssl_transport_security.cc:1511] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.       
E0311 13:58:55.402000000 31616 src/core/tsi/ssl_transport_security.cc:1511] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.       
E0311 13:58:57.076000000 31616 src/core/tsi/ssl_transport_security.cc:1511] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.       
E0311 13:58:59.639000000 31616 src/core/tsi/ssl_transport_security.cc:1511] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.       
Error connecting to Milvus: <MilvusException: (code=2, message=Fail connecting to server on <loadbalancer_external_ip>:19530. Timeout)>
Connection to Milvus closed.
E0311 13:59:03.277000000 31616 src/core/tsi/ssl_transport_security.cc:1511] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.  

[mahaveer08]

Deployed and connected to milvus db successfully!

Thanks for the help!

[guycohenzedek]

Can you share what was the final solution?
I'm trying to connect milvus-standalone(2.4) to MinIO with S3 bucket but got this:
[2024/06/23 08:13:16.010 +00:00] [WARN] [storage/minio_object_storage.go:139] ["failed to check blob bucket exist"] [bucket=<my_bucket>] [error="Head "https://minio:9000/<my_bucket>/": http: server gave HTTP response to HTTPS client"]

I changed the milvus.yaml file as described in documentation:
minio:

address: "s3.us-west-1.amazonaws.com"
port: 443
accessKeyID: <my_key>
secretAccessKey: <my_secret>
useSSL: true
bucketName: <my_bucket>

What is missing?

[xiaofan-luan]

Can you share what was the final solution? I'm trying to connect milvus-standalone(2.4) to MinIO with S3 bucket but got this: [2024/06/23 08:13:16.010 +00:00] [WARN] [storage/minio_object_storage.go:139] ["failed to check blob bucket exist"] [bucket=<my_bucket>] [error="Head "https://minio:9000/<my_bucket>/": http: server gave HTTP response to HTTPS client"]

I changed the milvus.yaml file as described in documentation: minio:

address: "s3.us-west-1.amazonaws.com" port: 443 accessKeyID: <my_key> secretAccessKey: <my_secret> useSSL: true bucketName: <my_bucket>

What is missing?

I think this is bacause you setup tls on client side but didn't enable tls on server side

[mahaveer08]

Feedback: Need a step to create a role to attach service account so that was bit confusing at the end!
Thanks @xiaofan-luan and @haorenfsa for inputs!

[haorenfsa]

Need a step to create a role to attach service account so that was bit confusing at the end!

Yes, this step is missing in the doc. We'll update it soon.

[haorenfsa]

tracking this in milvus-io/milvus-docs#2455

[mahaveer08]

Hi @xiaofan-luan and @haorenfsa I have one doubt regarding inserting data to milvus:

JSON

[
    {
      "id": 1,
      "name": "John Doe",
      "position": "Software Engineer",
      "department": "Engineering",
    },
    {
      "id": 2,
      "name": "John Doe",
      "position": "Data Scientist",
      "department": "Research",
      "salary": 90000
    },
    {
      "id": 3,
      "name": "Alice Johnson",
      "position": "Product Manager",
      "department": "Product Management",
      "salary": 100000
    },
    {
      "id": 4,
      "name": "Bob Brown",
      "position": "Marketing Specialist",
    }
  ]

this is the json I want to embed into milvus db so how does schema works here like I don't have constant schema's for each record? In need your input on this

[yhmo]

I think the key is to find a proper model to embed your data. But I am not an expert in this area.

[xiaofan-luan]

I think it's kind like based on your use case? what is the search gonna to be?
Any expression? Any keyword?

[mahaveer08]

Does querying on json data that stored in milvus it is difficult? Like I see no much documentation about json data storage and querying on them and expecting some result’s

[yhmo]

Querying on json data is not efficient in milvus.
Read this doc: https://milvus.io/docs/v2.3.x/json_data_type.md

[mahaveer08]

Got it thanks!

[xiaofan-luan]

id, name, postion and embeddding is fixed schema.
department and salary put into dynamic schema

[mahaveer08]

While building lambda function for pymilvus so the folder size is more than expected so do we have any way to reduce the same so that we can upload it to lambda?

[mahaveer08]

I am trying to build lambda using pymilvus package but the size of folder is 400MB so unable to reduce the size. What are the packages does support for pymilvus so the remaining packages I can remove.

[xiaofan-luan]

I am trying to build lambda using pymilvus package but the size of folder is 400MB so unable to reduce the size. What are the packages does support for pymilvus so the remaining packages I can remove.

pymilvus should be only 1.2 MB https://test.pypi.org/project/pymilvus/#files/

Are you saying you want to deploy milvus in a lambda function?

[mahaveer08]

No not milvus, I have lambda function where I am using pymilvus so need to build the zip file for that but size seems more than expected.

[xiaofan-luan]

you can submit a issue to pymilvus repo.
Maintainer there could help you to check