[Bug]: Unable to replicate data to target milvus ("the collection is not in the watch list")

[ckad-gr]

Current Behavior

No replication takes place between source and target milvus.

Expected Behavior

Any partition/collection being created and data added to them in source milvus should be replicated in target milvus.

Steps To Reproduce

1. Start target milvus with common.ttMsgEnabled: false
2. Start CDC with following config.

cdc-config.txt

3. Create a task in CDC like so:

curl -X POST http://localhost:8444/cdc -H "Content-Type: application/json" -d '{ "request_type": "create", "request_data": { "milvus_connect_param": { "host": "redacted-project.redacted-project-2", "port": 19530, "enable_tls": true, "username": "admin", "password": "PASSWORD", "connect_timeout": 10, "dial_config": { "server_name": "redacted-project.redacted-project-2", "server_pem_path": "/tmp/cert.pem" } }, "collection_infos": [ { "name": "*" } ] } }'

4. Create new partition in source milvus. Insert data into the new partition source milvus via attu. ("cdc_database" > "cdc_collection" > "cdc_partition")

5. Try pausing and resuming the CDC task just to check if that fixes anything.

Environment

Milvus: v2.4.17
Milvus-cdc: v2.0.0-rc4
compiled from git project, running inside docker image: docker.io/milvusdb/milvus-cdc@sha256:593da341ba7282583a9fed3f92f95d6a42e275231f7b6762f9524470e29f0568
With pulsar and etcd.

CDC is connected to the etcd and pulsar of source milvus.
CDC is using the etcd of source milvus under a different rootPath.

Anything else?

Logs:
logs.txt

The log line "the collection is not in the watch list" seems interesting.
I wonder if I'm misconfiguring things such that CDC is skipping all collections.

Note: "test" "actual_data" collection/db were deleted before starting this test.

[ckad-gr]

I've also tried using

    "collection_infos": [
      {
        "name": "cdc_collection"
      }
    ]

apart from just "name": "*"

[SimFG]

Because by default, collection info only synchronizes collections under the default db. You can try to create a collection under the default db.

[ckad-gr]

@SimFG thanks for the reply. Is there a way to synchronise collections under other databases yet ?