proposal: track changes of downloaded lists

[baldurh]

We would like to know if and when lists were updated.

Scenario: We need to screen all our customers when the lists change. We know when we last screened each customer. We would like to query watchman and get the date of not when the lists were last downloaded but when the lists actually changed. This way we can screen the customer only when needed.

[adamdecaf]

I've long thought about storing the hash of each list with the download stats. We don't store searches though as that usually falls to another service.

[baldurh]

Sorry for not being clear. I did not mean for watchman to store anything about the searches. We are already storing on our end when a customer was screened.

But yeah, storing the hash of the lists would allow us to achieve what we want 😊

[elovelan]

this may be too different a use case, but we've been talking internally about re-implementing the watches feature outside of Watchman. One optimization we've discussed is to not only hash the list, but hash each record in each list, so we only have to re-search against list entries that have changes.