Too many workflows at temporal

[nhdms]

Hi team,
I've been running peerdb for about half a day, but the number of "Continue as New" workflows keeps increasing continuously. Is there any problem here?
Thanks

[iamKunalGupta]

Can you let us know the PeerDB version? cc @heavycrystal

[heavycrystal]

what is your Sync Interval? You can find it in the status page of the mirror

[nhdms]

I'm using the default interval (60s) and latest version of peerdb (0.20.2) @heavycrystal @iamKunalGupta
Thanks

[heavycrystal]

A workflow should ContinueAsNew every interval, which is 60 seconds so this seems reasonable.
If you're concerned about too many workflows you can increase the interval of the mirror. We're also working on reducing the number of workflows needed overall and you should see improvements in upcoming releases

[nhdms]

@heavycrystal Thank you, after 1 day, my Mirror (postgresql to clickhouse) is stuck here.
I tried to decrease max_batch_size to 100000 and increase the timeout to 0, but nothing changed

[nhdms]

And the lag size keeps increasing, and it causes all other mirrors to get stuck :'(

[heavycrystal]

logs will be needed from the PeerDB containers to help debug this, it'll be easier to join our Slack for the same