Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming to postgresql is Extremely slow. #457

Open
B1Q opened this issue Sep 5, 2023 · 2 comments
Open

Streaming to postgresql is Extremely slow. #457

B1Q opened this issue Sep 5, 2023 · 2 comments

Comments

@B1Q
Copy link

B1Q commented Sep 5, 2023

I noticed that streaming is very very slow.

I'm running an Erigon Archive node locally

with this command
ethereumetl stream -e block,transaction,token_transfer,token,log --provider-uri http://127.0.0.1:8545 --output=postgresql+pg8000://user:[email protected]:5432/db--period-seconds 0 --max-workers 1000

It took 2 hours to index 15963 blocks. (recorded the time)
this is indexing old blocks starting from block 10,000,000
I am also running another one that's indexing the blockchain from the last block onwards (to keep database up to date)

Any advice on how to speed it up?

@medvedev1088
Copy link
Member

This could be useful https://github.com/blockchain-etl/ethereum-etl-postgres

@B1Q
Copy link
Author

B1Q commented Sep 5, 2023

This could be useful https://github.com/blockchain-etl/ethereum-etl-postgres

Thank you, I have another question:

I am trying to download the data locally, is that a possibility? I do not wish to use Cloud SQL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants