Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance nodes IBD #50

Merged
merged 8 commits into from
Oct 9, 2024
Merged

Conversation

GitGab19
Copy link
Collaborator

@GitGab19 GitGab19 commented Sep 5, 2024

This PR drastically improves the UX when using the tool on mainnet.
If it's the first time the tool is run on mainnet, the update-mainnet-chainstate.sh script will download the latest snapshot from our VPS, which serves a new snapshot every hour. It will do it also if more than 24 hours have passed since the last time the tool has been used in the mainnet (since it would take a lot of time to sync 3 pruned nodes together).

If we still want to support testnet3 (which I wouldn't), I think we should replicate this PR also for that since it takes a lot of time to sync it. Otherwise we can just support mainnet and testnet4 and open an issue to remove testnet3 from the options in the run-benchmarking-tool.sh and wherever it's mentioned.

Close #38

@GitGab19 GitGab19 force-pushed the enhance-nodes-ibd branch 2 times, most recently from d8b346d to da3051e Compare September 5, 2024 17:07
Copy link
Collaborator

@Shourya742 Shourya742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK. Just few nits.

How many buffer snapshots are we keeping on the server? For example, if it takes me about 2 hours to download a snapshot, the if condition if wget $BACKUP_URL -O $BACKUP_FILE && wget $BACKUP_HASH_URL -O $BACKUP_HASH_FILE; then is gonna download the whole backup_file first, then the backup_hash_url to verify its authenticity. If we’re storing a lot of buffer snapshots, that’s fine (as long as it doesn't overload the server). But if we’re only keeping a few, like the last 5 hours’ worth, it could be a problem.

containers-scripts/update-mainnet-chainstate.sh Outdated Show resolved Hide resolved
containers-scripts/update-mainnet-chainstate.sh Outdated Show resolved Hide resolved
docker-compose-config-a.yaml Show resolved Hide resolved
@Shourya742
Copy link
Collaborator

Getting some socket hung up issue:
Screenshot 2024-09-16 at 4 57 00 PM

@GitGab19
Copy link
Collaborator Author

Concept ACK. Just few nits.

How many buffer snapshots are we keeping on the server? For example, if it takes me about 2 hours to download a snapshot, the if condition if wget $BACKUP_URL -O $BACKUP_FILE && wget $BACKUP_HASH_URL -O $BACKUP_HASH_FILE; then is gonna download the whole backup_file first, then the backup_hash_url to verify its authenticity. If we’re storing a lot of buffer snapshots, that’s fine (as long as it doesn't overload the server). But if we’re only keeping a few, like the last 5 hours’ worth, it could be a problem.

Atm we're keeping last 3 hours. How many do you suggest to keep?

@GitGab19
Copy link
Collaborator Author

Getting some socket hung up issue: Screenshot 2024-09-16 at 4 57 00 PM

Was the SV1 part working in this case or not?
I'm not experiencing that

Copy link
Collaborator

@Shourya742 Shourya742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rt-ACK. If we can have the IBD downloaded once and then mount it as a volume for the template provider or node containers, that would be ideal. Everything else looks good.

Copy link
Collaborator

@Shourya742 Shourya742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK

commented code removal
Copy link
Collaborator

@Shourya742 Shourya742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK 30066b2. Snapshot are getting refreshed for new downloads

@GitGab19 GitGab19 merged commit 9da0e73 into stratum-mining:main Oct 9, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Better manage Bitcoin nodes setup and synchronization
2 participants