Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use state to avoid re-sending data to observers #554

Open
lgalabru opened this issue Apr 5, 2024 · 2 comments
Open

Use state to avoid re-sending data to observers #554

lgalabru opened this issue Apr 5, 2024 · 2 comments

Comments

@lgalabru
Copy link
Contributor

lgalabru commented Apr 5, 2024

When a chainhook-node is re-processing blocks (for whatever reason), it should avoid re-sending blocks to observers when they have been sent in the past.
Data from Redis could help.

@github-project-automation github-project-automation bot moved this to 🆕 New in DevTools Apr 5, 2024
@andresgalante andresgalante moved this from 🆕 New to 📋 Backlog in DevTools Apr 8, 2024
@andresgalante
Copy link
Member

We might need to wait after the end of April (BTC++ conf) to take on this one

@smcclellan smcclellan added this to the Production Reliability milestone Apr 11, 2024
@MicaiahReid
Copy link
Contributor

To be honest, I don't know exactly how this one should be approached. But here are some notes:

We already have this to some degree when a Chainhook node is restarted. When a predicate is in Streaming mode, we update the status of the predicate after every block, storing the last evaluated block. When a predicate is in Scanning mode, we update the status after every match between a block + predicate. With these values stored, whenever Chainhook is restarted, the service can load the predicates from redis, evaluate each predicate's status to determine the last block evaluated, and continue where it left off. This prevents a restart of the Chainhook node from re-sending data to observers.

This issue, however, is for another case (which happened recently). If a Stacks node has to be reset, say to chain tip - 100 blocks, each of those 100 blocks will have already been evaluated by Chainhook. However, when the block is received by the event observer there currently is no way to determine if the predicate has already been evaluated against that block (the Chainhook SDK does not have access to the redis store of predicate status). Furthermore, even if the predicate has a last_evaluated_block, in many cases (such as a reorg), we should still evaluate the block.

It would be nice to have some guarantee that a predicate will never be triggered twice from the same block, but I think it will be quite difficult and is currently handled for the general case. The recent events on the testnet that caused us to reset the chain will probably be very rare, so maybe we aren't as concerned about handling it.

I wouldn't say I recommend closing this issue at this point, but maybe it can be revisited in a few months

@smcclellan smcclellan removed this from the Production Reliability milestone Jun 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: 📋 Backlog
Development

No branches or pull requests

4 participants