Conversation
- Introduce exponential backoff with configurable retries for failed HTTP requests in the crawler. - Add automatic cleanup of meta table entries older than a configurable retention period. - Update README and CHANGELOG with new configuration details.
- Replaced `.unwrap()` across the codebase with proper error handling to prevent panics. - Improved logging for failed operations in CLI, database, crawlers, and API. - Updated error messaging for better debugging. - Enforced `unwrap_used = "deny"` in lint configuration. - Added detailed CHANGELOG updates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces significant improvements to error handling throughout the codebase, adds new configuration options for crawler retry logic and database maintenance, and improves documentation accordingly. The main focus is on replacing
.unwrap()calls with proper error handling, making the CLI and API more robust and user-friendly. Additionally, the crawler and database maintenance processes are now more configurable and resilient.Error Handling Improvements
Replaced all
.unwrap()calls with proper error handling across the CLI, API, and background tasks, ensuring that errors are logged with informative messages and the application exits gracefully when necessary. This affects database operations, file downloads, collector loading, and time parsing.Added a Clippy lint to deny the use of
unwrapin future code (Cargo.toml).Crawler and Database Maintenance Enhancements
Added retry logic with exponential backoff for failed HTTP requests in the crawler, configurable via
BGPKIT_BROKER_CRAWLER_MAX_RETRIESandBGPKIT_BROKER_CRAWLER_BACKOFF_MSenvironment variables.Implemented automatic cleanup of old meta table entries after each database update, with retention configurable via
BGPKIT_BROKER_META_RETENTION_DAYS.Documentation Updates
CHANGELOG.mdandREADME.mdto document new configuration options and summarize the error handling improvements and new features.