Add Section for Version Tracking, Restart Behavior, and Backfilling #769
+80
−5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dermanyang
requested review from
a team,
gregnazario,
hariria and
igor-p
as code owners
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
| The `IndexerProcessorConfig`, `ProcessorStatusSaver` and `get_starting_version()` can all be modified to allow for convenient backfills. Without these changes, it is difficult to run a live processor at the latest transaction version as well as a backfill processor. | ||
|
|
||
| ### Updates to Config | ||
| [Add an additional field](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/config/indexer_processor_config.rs#L44) on your `IndexerProcessorConfig` for a `BackfillConfig` which contains a single field, `backfill_alias` . The introduction of this field will be used to determine whether the processor is a backfill processor or regular one. |
There was a problem hiding this comment.
Remove the extra spaces around the period in backfill_alias .
Spotted by Graphite Reviewer
Is this helpful? React 👍 or 👎 to let us know.
rtso
approved these changes
Jan 15, 2025
| To enable backfilling, `IndexerProcessorConfig`, `ProcessorStatusSaver` and `get_starting_version()` need some updates. Without these changes, it is difficult to run a live processor at the latest transaction version as well as a backfill processor. | ||
|
|
||
| ### Updates to Config | ||
| [Add an additional field](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/config/indexer_processor_config.rs#L44) on your `IndexerProcessorConfig` for a `BackfillConfig` which contains a single field, `backfill_alias`. The introduction of this field will be used to determine whether the processor is a backfill processor or regular one. |
There was a problem hiding this comment.
I've been linking to the example repo event processor implementation since it's an easier starting point for beginners. We could link to the core processors if they need more examples.
|
|
||
| ### Updates to get_starting_version | ||
| Add a [statement](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/utils/starting_version.rs#L143) in your `get_starting_version` method to query the `backfill_processor_status_table` when the `BackfillConfig` field is present in `IndexerProcessorConfig` . | ||
|
|
There was a problem hiding this comment.
[Re: line +65]
nit: can we add a small section on how to update the `config.yaml` to run the processor in backfill mode?
See this comment inline on Graphite.
nit: can we add a small section on how to update the `config.yaml` to run the processor in backfill mode?
See this comment inline on Graphite.
| ``` | ||
|
|
||
| ### Backfill Processor Status Table | ||
| To avoid write conflicts with the regular `processor_status` table, introduce a `backfill_processor_status_table` that will use `backfill_alias` as the primary key instead of `processor_name`. Expand on this [implementation](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/db/common/models/backfill_processor_status.rs). This model introduces a new state, `BackfillStatus`, which is either `InProgress` or `Complete` which will determine the backfilling restart behavior. |
There was a problem hiding this comment.
Might also help to explain why it's important to avoid write conflicts - usually we run head and backfill processors at the same time so that data still gets indexed at head.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Checklist
pnpm fmt?pnpm lint?