Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GRPC Ingest #183

Draft
wants to merge 61 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
ea5379e
fmt
fanatid Nov 20, 2023
8b12316
clippy
fanatid Nov 20, 2023
974c90f
ci: add lock, fmt, clippy checks
fanatid Nov 20, 2023
788973e
include migration to workspace
fanatid Nov 20, 2023
7a870c2
include das_api to workspace
fanatid Nov 20, 2023
7e3f9ef
add global clippy lints
fanatid Nov 20, 2023
aa916a4
use workspace
fanatid Nov 20, 2023
033873d
add crate program_transformers
fanatid Nov 22, 2023
1008f86
nft_ingester: use program_transformers crate
fanatid Nov 23, 2023
83b171e
remove not used deps
fanatid Nov 23, 2023
9d8031e
remove AccountInfo
fanatid Nov 23, 2023
096616d
remove plerkle from program_transformers
fanatid Nov 24, 2023
ef0b0e9
nft_ingester2: grpc2redis
fanatid Nov 27, 2023
076bffa
add redis streaming for ingester
fanatid Nov 28, 2023
d8673cf
create pg pool
fanatid Nov 29, 2023
c93adff
parse incoming message from redis
fanatid Nov 29, 2023
53fa481
add force shutdown with signals
fanatid Nov 30, 2023
11b4151
insert download metadata tasks
fanatid Dec 1, 2023
8fca2c2
download-metadata subtask
fanatid Dec 2, 2023
fcc18d6
refactor: rename nft_ingest2
kespinola Apr 10, 2024
294fa1a
fix: applying account and transction filters to grpc subscription req…
kespinola Apr 15, 2024
98a7e4e
refactor: take out multiple connections to dragonmouth and then have …
kespinola May 2, 2024
997d515
refactor: switch to accepting multiple dragonmouth endoints to mimic …
kespinola May 3, 2024
75c4115
Merge branch 'main' of github.com:rpcpool/digital-asset-rpc-infrastru…
kespinola May 9, 2024
67cc21e
fix: clippy errors
kespinola May 9, 2024
5f08298
fix: processes transactions for a tree sequentially to make sure ther…
kespinola May 15, 2024
0eced5d
refactor: use program transform in the account backfill no queuing in…
kespinola May 16, 2024
138bf2e
refactor: move bubble backfill to a lib so can be used by other proje…
kespinola May 30, 2024
d80e7b8
Merge remote-tracking branch 'origin' into grpc-ingest
kespinola Jul 19, 2024
2cc3411
chor: refactor grpc command to use topograph to clean up the control …
kespinola Jul 30, 2024
81f5d71
chore: switch to run_v2 for ingest which handles ingesting accounts a…
kespinola Aug 9, 2024
10c46d5
Merge remote-tracking branch 'origin' into grpc-ingest
kespinola Aug 10, 2024
a151630
chore: handle metadata json within the ingest command
kespinola Aug 12, 2024
f398880
test: metadata json stream processing
kespinola Aug 12, 2024
70eef04
feat: set num threads for topograph. report on postgres and redis in …
kespinola Aug 14, 2024
77a98f6
chore: switch to topograph for thread management. create separate ing…
kespinola Aug 20, 2024
0db0612
fix: config and ack
kespinola Aug 22, 2024
32f9852
hack: disable pending processing while see why messages arent being d…
kespinola Aug 22, 2024
bab4678
fix: exit early if slot matches metadata json download and reindex is…
kespinola Aug 22, 2024
0c4f872
fix: picking dangling pending messages before reading new
kespinola Aug 22, 2024
eb75ca3
fix: ingest stream shutdown and requesting snapshots
kespinola Aug 23, 2024
3213080
fix: processing pending. add metrics for total workers.
kespinola Aug 23, 2024
7f3b391
fix: bubble backfill transform. no op out of download metadata info.
kespinola Aug 26, 2024
4c3f649
Add force flag to TreeWorkerArgs for complete reindexing (#148)
kevinrodriguez-io Sep 13, 2024
4e3c076
refactor: drop exculsion clauses on cl_items to allow for reindexing …
kespinola Sep 16, 2024
c789b52
feat: replay bubblegum transactions based on what is in cl_audits_v2
kespinola Sep 26, 2024
45a5dcf
fix: order bubblegum backfill instructions by seq (#158)
kespinola Oct 2, 2024
f4744c0
include back seq checks
kespinola Oct 7, 2024
1afee45
Remove enforcing seq in bubblegum backfill
kespinola Oct 7, 2024
18411a9
Replay single tree with targeted seq replay (#159)
kespinola Oct 8, 2024
110f71c
Support Account Snapshot Flushes (#154)
kespinola Oct 9, 2024
118baec
fix: keep grpc alive by sending using ping-pong (#160)
Nagaprasadvr Oct 9, 2024
01bbe99
Only Time Based Flushes (#161)
kespinola Oct 9, 2024
cde4594
delete and recreate the consumer (#162)
kespinola Oct 11, 2024
0005b3e
Type out redis message handlers (#163)
kespinola Oct 12, 2024
d092ad8
skip reading new messages if the ack buffer is full. track running ta…
kespinola Oct 13, 2024
6f158f3
only push handler job for ingest if capacity (#165)
kespinola Oct 14, 2024
a347066
Only throttle based on the task semaphore (#166)
kespinola Oct 14, 2024
188c878
Remove topograph from ingest command (#167)
kespinola Oct 14, 2024
e15172f
Configure database connection idle timeout and max lifetime (#169)
kespinola Oct 21, 2024
fdbf03a
Add single NFT fetch and backfill feature to ops (#170)
Nagaprasadvr Oct 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions .github/workflows/build-api.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ on:
workflow_dispatch:
push:
branches:
- 'main'
- "main"

env:
CARGO_TERM_COLOR: always
CARGO_TERM_COLOR: always

jobs:
build-api:
Expand All @@ -33,15 +33,15 @@ jobs:
- name: set build cache
uses: actions/cache@v3
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
digital-asset-rpc-infrastructure/target/
key: ${{ matrix.os }}_digital-asset-rpc-infrastructure_${{ hashFiles('digital-asset-rpc-infrastructure/Cargo.lock') }}
restore-keys: |
${{ matrix.os }}_digital-asset-rpc-infrastructure
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
digital-asset-rpc-infrastructure/target/
key: ${{ matrix.os }}_digital-asset-rpc-infrastructure_${{ hashFiles('digital-asset-rpc-infrastructure/Cargo.lock') }}
restore-keys: |
${{ matrix.os }}_digital-asset-rpc-infrastructure

- name: build digital asset rpc infra
run: cargo build --verbose --release
Expand All @@ -54,7 +54,7 @@ jobs:
mv target/release/migration target/release/migration22
mv target/release/das_api target/release/das_api22

# This steps can be omited to save space, are mostly in place to validate binaries (manually) and path to them
# This steps can be omited to save space, are mostly in place to validate binaries (manually) and path to them
# Omiting this will save on storage consumption on the account
- name: Publish artifact
if: matrix.os == 'ubuntu-22.04'
Expand Down
20 changes: 10 additions & 10 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ on:
pull_request:
push:
branches:
- 'main'
- "main"
workflow_dispatch:

env:
CARGO_TERM_COLOR: always
CARGO_TERM_COLOR: always

jobs:
test:
Expand All @@ -25,20 +25,20 @@ jobs:
- name: set build cache
uses: actions/cache@v3
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
digital-asset-rpc-infrastructure/target/
key: cargo-${{ hashFiles('**/Cargo.lock') }}-0001
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
digital-asset-rpc-infrastructure/target/
key: cargo-${{ hashFiles('**/Cargo.lock') }}-0001

# Cargo.lock
- name: Check lock file
run: |
cargo tree
git checkout Cargo.lock
cargo tree --frozen
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yellowstone crates are referenced using git so require network fetch to retrieve which violates frozen flag.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cargo tree
# fmt
- name: Check fmt
Expand Down
9 changes: 5 additions & 4 deletions Builder.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
FROM rust:1.76-bullseye AS builder
FROM rust:1.75-bullseye AS builder
RUN apt-get update -y && \
apt-get install -y build-essential make git
apt-get install -y build-essential make git

RUN mkdir /rust
RUN mkdir /rust/bins
COPY Cargo.toml /rust
COPY core /rust/core
COPY backfill /rust/backfill
COPY das_api /rust/das_api
COPY digital_asset_types /rust/digital_asset_types
COPY integration_tests /rust/integration_tests
Expand All @@ -19,7 +20,7 @@ COPY blockbuster rust/blockbuster
WORKDIR /rust
RUN --mount=type=cache,target=/rust/target,id=das-rust \
cargo build --release --bins && cp `find /rust/target/release -maxdepth 1 -type f | sed 's/^\.\///' | grep -v "\." ` /rust/bins

FROM rust:1.75-slim-bullseye as final
COPY --from=builder /rust/bins /das/
CMD echo "Built the DAS API bins!"
Loading
Loading