fix: add event index to transaction events #417

janniks · 2023-09-18T13:48:17Z

closes return event_index in Stacks transaction event metadata #387
adds a position struct to the event and renames the event to payload (with enum type)
updates occurences.json to include the expected serialized position for testing

### Description Improves how we handle a restart of `chainhook service` while predicates are scanning/streaming. Here are the cases we now handle: 1. Predicates that were in `scanning` status when Chainhook was terminated will resume scanning starting from their `last_evaluated_block_height`. *Note: because we only save predicate status every 10 scans, we could end up re-emiting matches on a resetart* 2. Predicates that were in `new` status when Chainhook was terminated will start scanning at the predicate's `start_block` 3. Predicates that were in `streaming` status will _return_ to a `scanning` status, starting at `last_evaluated_block_height` to catch up on the missed blocks. Note, the `number_of_blocks_to_scan` is set to 0 for this temporary catch-up, as it's difficult to compute the number of remaining blocks in the context of this change 4. If predicates were passed in at startup, we also register those to begin scanning, which previously didn't happen 5. We now allow passing in a predicate at startup _and_ registering additional predicates with the predicate registration server. This means that if you use the same startup predicate repeatedly, it will already be saved in redis and _not_ be reloaded. Fixes: #298, fixes #390, fixes #402, fixes #403 --- ### Checklist - [x] All tests pass - [ ] Tests added in this PR (if applicable)

…ion-event-metadata

MicaiahReid

Thanks, @janniks!!

MicaiahReid

I've discovered some ways where this may cause breakages, I need to do some investigation.

MicaiahReid

Okay, it looks like it should be good!

…ion-event-metadata

lgalabru

Thanks @janniks! We already have the struc StacksTransactionPosition in place, indicating the position of a transaction in the block or in the microblock, could we use this instead?
Apologies for the latency on this review.

janniks · 2023-10-31T16:55:38Z

Oh yeah sure, sorry for missing that 👍

janniks · 2023-11-06T18:03:43Z

The new positions are referring to events not blocks. Should we create an enum type that mixes blocks and events (might be funky when matching). Or add an additional enum layer EventPosition (but that also sounds complected)?

…ion-event-metadata

MicaiahReid · 2024-01-11T14:58:06Z

@lgalabru I don't know if I agree - Jannik's change is adding a struct for an event's position in a transaction, while StacksTransactionPosition represents a transaction's position within a block.

I guess we could abstract that position enum to be more general, from:

pub enum StacksTransactionPosition {
    AnchorBlock(AnchorBlockPosition),
    MicroBlock(MicroBlockPosition),
}

to:

pub enum Position {
    TransactionPositionInAnchorBlock(AnchorBlockPosition),
    TransactionPositionInMicroBlock(MicroBlockPosition),
    EventPositionInTransaction(u64)
}

but that feel's clunky. Let me know if we're misunderstanding what you meant here!

lgalabru

My apologies for the latency on this PR, yes that makes sense to me.
Thank you for your help @janniks!

MicaiahReid · 2024-02-12T19:48:00Z

Closing in favor of #495

@janniks

(this PR is just cherry-picking the relevant commits from @janniks in PR #417, which I apparently mucked up with some bad merging) - closes #387 - adds a position struct to the event and renames the event to payload (with enum type) - updates occurences.json to include the expected serialized position for testing --- **Note**: Any users running a Chainhook node (i.e. `chainhook service start ...`) will not have transaction event positions stored in the database, so all event position will have a value of: ``` JSON "position": { "index": 0 } ``` To rebuild the database with this data filled in: - Upgrade to the latest version of Chainhook - Delete the `stacks.rocksdb` folder inside the `working_dir` set in your `Chainhook.toml` - Rerun Chainhook This will rebuild your Stacks database from scratch. --------- Co-authored-by: janniks <[email protected]>

@janniks

(this PR is just cherry-picking the relevant commits from @janniks in PR #417, which I apparently mucked up with some bad merging) - closes #387 - adds a position struct to the event and renames the event to payload (with enum type) - updates occurences.json to include the expected serialized position for testing --- **Note**: Any users running a Chainhook node (i.e. `chainhook service start ...`) will not have transaction event positions stored in the database, so all event position will have a value of: ``` JSON "position": { "index": 0 } ``` To rebuild the database with this data filled in: - Upgrade to the latest version of Chainhook - Delete the `stacks.rocksdb` folder inside the `working_dir` set in your `Chainhook.toml` - Rerun Chainhook This will rebuild your Stacks database from scratch. --------- Co-authored-by: janniks <[email protected]>

## [1.3.1](v1.3.0...v1.3.1) (2024-02-14) ### Bug Fixes * add event index to transaction events ([#495](#495)) ([d67d1e4](d67d1e4)), closes [#417](#417) [#387](#387) * correctly determine PoX vs PoB block commitments ([#499](#499)) ([50dd26f](50dd26f)), closes [#496](#496)

## [1.3.1](hirosystems/chainhook@v1.3.0...v1.3.1) (2024-02-14) ### Bug Fixes * add event index to transaction events ([#495](hirosystems/chainhook#495)) ([2d2760d](hirosystems/chainhook@2d2760d)), closes [#417](hirosystems/chainhook#417) [#387](hirosystems/chainhook#387) * correctly determine PoX vs PoB block commitments ([#499](hirosystems/chainhook#499)) ([6816e76](hirosystems/chainhook@6816e76)), closes [#496](hirosystems/chainhook#496)

MicaiahReid added 30 commits August 24, 2023 15:19

spelling fix in tests

aa30d42

variable rename

96fb8dd

update status for first scan

4457533

update StreamingData status

0f8ed57

missed rename

aea9e15

add more info to predicate status updates

1832ec8

doc comments

9eec034

remove changes to Interrupted status

73a9c0f

return error details to user for failed hook actions

9741167

add predicate streaming status update for bitcoin

3208481

augment LIST endpoint; merge helpers

78592ba

variable/status renames

7742043

check end_block when evaluating blocks

dc2cbd9

set expired status

dc63688

missed updates in setting expired status

8b5fa75

fix blockheight after scan

3166833

simplify fix for blockheight after scan

3b47b33

fix expiration tracking

4fd2ebb

revert stuff moved to other PR

ba775a5

allow expired => streaming/scanning status update

6295b86

remove unused var

6530b6a

add doc images

726024e

link to doc image

e68b095

expire blocks once confirmed by chain

eb08eda

create db on readonly connect if not exist

636f510

add some serde derivations

8ab1dd0

only enable predicate if not expired

53080f7

unimplemented => noop

c7b4d7a

add tests for chainhook service get status api

01ca09b

MicaiahReid added 3 commits September 22, 2023 15:35

chore: update predicate status format for easier parsing

33e8378

fix tests for serde change

a6b43d5

remove debug logging

44fedf2

janniks force-pushed the 387-return-event_index-in-stacks-transaction-event-metadata branch from 02945dd to 5d359ba Compare September 26, 2023 18:02

janniks marked this pull request as ready for review September 26, 2023 18:03

janniks and others added 3 commits October 2, 2023 20:45

fix: add event index to transaction events

aa1961d

fix: update tests

697baae

rebase fixes

0578ce5

MicaiahReid force-pushed the 387-return-event_index-in-stacks-transaction-event-metadata branch from 5d359ba to 0578ce5 Compare October 3, 2023 16:47

Merge branch 'develop' into 387-return-event_index-in-stacks-transact…

8469b83

…ion-event-metadata

MicaiahReid approved these changes Oct 3, 2023

View reviewed changes

MicaiahReid suggested changes Oct 5, 2023

View reviewed changes

MicaiahReid approved these changes Oct 5, 2023

View reviewed changes

Merge branch 'develop' into 387-return-event_index-in-stacks-transact…

b02d35c

…ion-event-metadata

lgalabru had a problem deploying to Development-mainnet October 23, 2023 15:40 — with GitHub Actions Failure

lgalabru had a problem deploying to Development-testnet October 23, 2023 15:40 — with GitHub Actions Failure

lgalabru suggested changes Oct 23, 2023

View reviewed changes

Merge branch 'develop' into 387-return-event_index-in-stacks-transact…

32495f4

…ion-event-metadata

janniks requested a review from lgalabru February 12, 2024 17:37

lgalabru approved these changes Feb 12, 2024

View reviewed changes

MicaiahReid mentioned this pull request Feb 12, 2024

fix: add event index to transaction events #495

Merged

MicaiahReid closed this Feb 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: add event index to transaction events #417

fix: add event index to transaction events #417

janniks commented Sep 18, 2023 •

edited

Loading

MicaiahReid left a comment

MicaiahReid left a comment

MicaiahReid left a comment

lgalabru left a comment

janniks commented Oct 31, 2023

janniks commented Nov 6, 2023 •

edited

Loading

MicaiahReid commented Jan 11, 2024

lgalabru left a comment

MicaiahReid commented Feb 12, 2024

fix: add event index to transaction events #417

fix: add event index to transaction events #417

Conversation

janniks commented Sep 18, 2023 • edited Loading

MicaiahReid left a comment

Choose a reason for hiding this comment

MicaiahReid left a comment

Choose a reason for hiding this comment

MicaiahReid left a comment

Choose a reason for hiding this comment

lgalabru left a comment

Choose a reason for hiding this comment

janniks commented Oct 31, 2023

janniks commented Nov 6, 2023 • edited Loading

MicaiahReid commented Jan 11, 2024

lgalabru left a comment

Choose a reason for hiding this comment

MicaiahReid commented Feb 12, 2024

janniks commented Sep 18, 2023 •

edited

Loading

janniks commented Nov 6, 2023 •

edited

Loading