mc diff error : "Unable to list comparison retrying.. XML syntax error on line 2: expected attribute name in element"

[boomballa]

Expected behavior

After performing data migration using mc mirror, you want to perform data validation properly via mc diff.

Actual behavior

In fact, at the mc diff step, an error is thrown all the time and data validation is not possible

debug.log

Steps to reproduce the behavior

1. mc alias list info

minio-online-cmpri:~$ mc alias list
cm-minio
  URL       :  http://192.168.50.50:8380
  AccessKey : minioadmin
  SecretKey : xxxxxxxxxx
  API       : s3v4
  Path      : auto
  Src       : /home/dba/.mc/config.json


rich-media-resource-16
  URL       : http://192.168.56.16:9091
  AccessKey : xxxxxxxxxx
  SecretKey : xxxxxxxxxx
  API       : s3v4
  Path      : auto
  Src       : /home/dba/.mc/config.json

2. Data validation after completion of data migration
   execute a command: mc diff --debug rich-media-resource-16/rich-media-resource cm-minio/rich-media-resource
3. After about ten seconds of execution, it outputs an error

mc: <ERROR> Unable to list comparison retrying.. XML syntax error on line 2: expected attribute name in element
 (1) difference.go:244 cmd.differenceInternal(..) Tags: [http://192.168.56.16:9091/rich-media-resource/, http://192.168.50.50:8380/rich-media-resource/]
 (0) client-s3.go:2416 cmd.(*S3Client).listRecursiveInRoutine(..)
 Commit:1681e4497c09 | Release-Tag:RELEASE.2024-11-21T17-21-54Z | Host:mongodb10-online-cmpri.gy.ntes | OS:linux | Arch:amd64 | Lang:go1.23.4 | Mem:9.9 MiB/33 MiB | Heap:9.9 MiB/22 MiB

Detailed execution debug log in the attached debug.log 。

mc --version

mc version RELEASE.2024-11-21T17-21-54Z (commit-id=1681e4497c09d7438a34e846f76dbde972ab7daf)
Runtime: go1.23.4 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html

System information

Linux minio10-online 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[jiuker]

Could you update the file to play?
Then we can see if that happend in play?
Use mc cp file play/bucket/path1 and mc cp file play/bucket/path2 to upload, then we can use diff

[boomballa]

Could you update the file to play? Then we can see if that happend in play? Use mc cp file play/bucket/path1 and mc cp file play/bucket/path2 to upload, then we can use diff

shell:~$ mc cp debug.log  play/bucket/test1/
/home/dba/debug.log:                       61.29 KiB / 61.29 KiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 42.88 KiB/s 1sshell:~$ mc cp debug.log  play/bucket/test2/
/home/dba/debug.log:                       61.29 KiB / 61.29 KiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 34.31 KiB/s 1sshell:~$ mc diff --debug play/bucket/test1/ play/bucket/test2/
mc: <DEBUG> GET /bucket/?location= HTTP/1.1
Host: play.min.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.77 mc/RELEASE.2024-11-21T17-21-54Z
Accept-Encoding: zstd,gzip
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20250305/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20250305T115528Z

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 137
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 05 Mar 2025 11:55:29 GMT
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: 3e996b2f640d7e065d3a5c4e39a5538cefb82e3e77771990265e4698d8681eac
X-Amz-Request-Id: 1829E619C69BCEF9
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 3
X-Ratelimit-Remaining: 3
X-Xss-Protection: 1; mode=block

mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Let's Encrypt
mc: <DEBUG>  >> Expires: 2025-04-17 05:49:37 +0000 UTC
mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Internet Security Research Group
mc: <DEBUG>  >> Expires: 2027-03-12 23:59:59 +0000 UTC
mc: <DEBUG> Response Time:  571.806935ms

mc: <DEBUG> GET /bucket/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&max-keys=1&prefix=test1%2F HTTP/1.1
Host: play.min.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.77 mc/RELEASE.2024-11-21T17-21-54Z
Accept-Encoding: zstd,gzip
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20250305/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250305T115529Z

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 626
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 05 Mar 2025 11:55:29 GMT
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: 3e996b2f640d7e065d3a5c4e39a5538cefb82e3e77771990265e4698d8681eac
X-Amz-Request-Id: 1829E619D1CF4C1B
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 3
X-Ratelimit-Remaining: 3
X-Xss-Protection: 1; mode=block

mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Let's Encrypt
mc: <DEBUG>  >> Expires: 2025-04-17 05:49:37 +0000 UTC
mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Internet Security Research Group
mc: <DEBUG>  >> Expires: 2027-03-12 23:59:59 +0000 UTC
mc: <DEBUG> Response Time:  188.824543ms

mc: <DEBUG> GET /bucket/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&max-keys=1&prefix=test2%2F HTTP/1.1
Host: play.min.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.77 mc/RELEASE.2024-11-21T17-21-54Z
Accept-Encoding: zstd,gzip
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20250305/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250305T115529Z

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 626
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 05 Mar 2025 11:55:29 GMT
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: 3e996b2f640d7e065d3a5c4e39a5538cefb82e3e77771990265e4698d8681eac
X-Amz-Request-Id: 1829E619DD1A506B
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 3
X-Ratelimit-Remaining: 3
X-Xss-Protection: 1; mode=block

mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Let's Encrypt
mc: <DEBUG>  >> Expires: 2025-04-17 05:49:37 +0000 UTC
mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Internet Security Research Group
mc: <DEBUG>  >> Expires: 2027-03-12 23:59:59 +0000 UTC
mc: <DEBUG> Response Time:  188.118997ms

mc: <DEBUG> GET /bucket/?delimiter=&encoding-type=url&fetch-owner=true&list-type=2&metadata=true&prefix=test1%2F HTTP/1.1
Host: play.min.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.77 mc/RELEASE.2024-11-21T17-21-54Z
Accept-Encoding: zstd,gzip
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20250305/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250305T115529Z

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 710
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 05 Mar 2025 11:55:29 GMT
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: 3e996b2f640d7e065d3a5c4e39a5538cefb82e3e77771990265e4698d8681eac
X-Amz-Request-Id: 1829E619E8A8CB4F
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 3
X-Ratelimit-Remaining: 3
X-Xss-Protection: 1; mode=block

mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Let's Encrypt
mc: <DEBUG>  >> Expires: 2025-04-17 05:49:37 +0000 UTC
mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Internet Security Research Group
mc: <DEBUG>  >> Expires: 2027-03-12 23:59:59 +0000 UTC
mc: <DEBUG> Response Time:  193.615305ms

mc: <DEBUG> GET /bucket/?delimiter=&encoding-type=url&fetch-owner=true&list-type=2&metadata=true&prefix=test2%2F HTTP/1.1
Host: play.min.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.77 mc/RELEASE.2024-11-21T17-21-54Z
Accept-Encoding: zstd,gzip
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20250305/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250305T115529Z

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 710
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 05 Mar 2025 11:55:30 GMT
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: 3e996b2f640d7e065d3a5c4e39a5538cefb82e3e77771990265e4698d8681eac
X-Amz-Request-Id: 1829E619F40837BC
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 3
X-Ratelimit-Remaining: 3
X-Xss-Protection: 1; mode=block

mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Let's Encrypt
mc: <DEBUG>  >> Expires: 2025-04-17 05:49:37 +0000 UTC
mc: <DEBUG> TLS Certificate found:
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Internet Security Research Group
mc: <DEBUG>  >> Expires: 2027-03-12 23:59:59 +0000 UTC
mc: <DEBUG> Response Time:  383.585227ms

@jiuker Thank you for your reply, I uploaded a file of about 65kb as you described and experimented with it and it doesn't look like there's anything wrong with it.

What I notice now is that the two s3 stores I examined before don't have the same region, where my minio I didn't set the region, and the s3 store region on the target is cn-ease-1, which I think defaults to us-ease-1 on my side, does the configuration of this piece affect the checksum?

Also, I've tried adding the region info for each of the two alias in the mc config file, but it doesn't seem to work.

[klauspost]

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 04 Mar 2025 07:15:25 GMT
Vary: Origin,Access-Control-Request-Method,Access-Control-Request-Headers
X-Amz-Request-Id: ky0AAK6UBLk8iCkY
X-Log: v4mget;dirCount:0 itemCount:1000 fdelCount:0;rs1.ls:18;rs1.confirmId:69;RSF:91;RSF:92;lb.v2.2/10.44.56.17:7913:92;S3APIv2:107
X-Reqid: ky0AAK6UBLk8iCkY

This response is not from a MinIO server. What is it from?

[boomballa]

mc: <DEBUG> HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 04 Mar 2025 07:15:25 GMT
Vary: Origin,Access-Control-Request-Method,Access-Control-Request-Headers
X-Amz-Request-Id: ky0AAK6UBLk8iCkY
X-Log: v4mget;dirCount:0 itemCount:1000 fdelCount:0;rs1.ls:18;rs1.confirmId:69;RSF:91;RSF:92;lb.v2.2/10.44.56.17:7913:92;S3APIv2:107
X-Reqid: ky0AAK6UBLk8iCkY

This response is not from a MinIO server. What is it from?

@klauspost Hello, thanks for your reply, my target end is an object storage service named kodo, also s3 protocol, what I am currently doing is to migrate the data from kodo to minio, the number of files can be seen in the dashboard, but I need to ensure the integrity of the data so there will be a mc diff verifying the files of this need.