Incremental replication method broken

[eldhosejohn]

I could not put a replication key when the replication method is INCREMENTAL. It is throwing keyError. While debugging I found only value it can take is 'replication_key'

[menzenski]

@eldhosejohn can you share more about the use case you have, and the behavior you'd like to see? Do you want to set it hte replication key to updatedAt or something similar?

[mw243270]

@menzenski Along these same lines I am trying to fully sync a MongoDB tap with a target and would like to use an "updatedOn". However, that replication key would need to be configurable per each collection as they could be different. In order to propagate all insert, updates, and deletes from the tap to destination, would you suggest running 1 config with INCREMENTAL mode to capture the inserts and updates and 1 config in LOG_BASED to get the deletes? I am trying to understand the workflow to get all changes to the target. Thanks!

[menzenski]

@eldhosejohn @mw243270 what sort of replication key are you wanting to use? Is it a MongoDB Date type?

[mw243270]

@menzenski I think pure CDC approach of watching the MongoDB oplog for all operations types (insert/update/delete) makes the most sense. That way there's no need to specify any replication key from the data to determine what record has changed. In an ideal world, the tap configured in LOG_BASED would recognize (based on state) it had never been run before and would trigger a full load of the collection(s). Subsequent runs would detect changes from oplog since the last run (based on state) and either insert/update or set the deleted flag to true in the target.