Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use parity-db in current Dockerfiles #455

Merged
merged 11 commits into from
Nov 30, 2023
Merged

Use parity-db in current Dockerfiles #455

merged 11 commits into from
Nov 30, 2023

Conversation

kayabaNerve
Copy link
Member

@kayabaNerve kayabaNerve commented Nov 26, 2023

The motivation for redb was to remove the multiple rocksdb compile times from CI.

@kayabaNerve kayabaNerve added improvement This could be better CI/CD Continuous integration workflows labels Nov 26, 2023
@kayabaNerve
Copy link
Member Author

This has massively faster CI times, yet I believe some services create TXNs while a TXN already exists, causing deadlocks with the redb backend :/

subxt also consistently fails for 4 specific tests with a websocket connectivity error? I have no idea why that's happening (a debug assertion, some really specific performance commentary...).

This PR likely should be broken into a few smaller PRs that let is triage individual changes.

@kayabaNerve kayabaNerve changed the title Use redb and debug builds in current Dockerfiles Use redb in current Dockerfiles Nov 29, 2023
@kayabaNerve
Copy link
Member Author

Rebased on top of #462.

@kayabaNerve
Copy link
Member Author

While I want to see if this PR's CI runs work (except coordinator and full stack), coordinator will always fundamentally fail. The tributary manages its own TXNs so we will always have nested TXNs.

sled? It hasn't had a release in a while, so we'd have to go to marble or it as a git dependency.

We could use the MemDB? AgateDB, by tikv? ParityDB, which is a patricia merkle trie yet won't incur C compile times?

It still has much better compile times yet doesn't block when creating multiple
transactions. It also is actively maintained and doesn't grow our tree. The MPT
aspects are irrelevant.
@kayabaNerve kayabaNerve changed the title Use redb in current Dockerfiles Use parity-db in current Dockerfiles Nov 30, 2023
@kayabaNerve
Copy link
Member Author

The prior full stack tests failed with the following.

2023-11-30T06:49:27.5628124Z [2023-11-30T06:41:01Z TRACE serai_coordinator] processor message effected transaction 95eb7054a83861413a3c66105e3847ba8c7613888998178449ffff33e4016eb7 Transaction::DkgCommitments { attempt: 0, signer: "e2f2ae0a6abc4e71a884a961c500515f58e30b6aa582dd8db6a65945e08d2d76", .. }
2023-11-30T06:49:27.5629363Z [2023-11-30T06:41:01Z TRACE serai_coordinator] getting next nonce for Tributary TX in response to processor message
2023-11-30T06:49:27.5630370Z [2023-11-30T06:41:01Z DEBUG serai_coordinator] publishing transaction e37b1ae87eb737b75af907d603a5d60c59298cbae7f5c8f036bcca1c1d190782
2023-11-30T06:49:27.5630879Z thread 'tokio-runtime-worker' panicked at coordinator/src/main.rs:167:17:
2023-11-30T06:49:27.5631858Z created an invalid transaction: InvalidSignature

I have absolutely no clue how this happened. The DB being corrupt should not have this effect. My only idea is DB Transaction::get being broken led to a genesis hash mismatch? That's the only piece of contextual data in the signing process.

@kayabaNerve kayabaNerve merged commit b823413 into develop Nov 30, 2023
19 checks passed
@kayabaNerve
Copy link
Member Author

This saves 20-25 minutes in the CI.

@kayabaNerve kayabaNerve deleted the redb-ci branch November 30, 2023 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI/CD Continuous integration workflows improvement This could be better
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant