branch-4.0: [Opt](warmup) Limit the download rate of warmup task (#60180)#60841
Open
bobhan1 wants to merge 1 commit intoapache:branch-4.0from
Open
branch-4.0: [Opt](warmup) Limit the download rate of warmup task (#60180)#60841bobhan1 wants to merge 1 commit intoapache:branch-4.0from
bobhan1 wants to merge 1 commit intoapache:branch-4.0from
Conversation
Summary This PR adds rate limiting support for warmup download tasks in file cache and introduces a generic DEFINE_ON_UPDATE macro for config update callbacks. Key changes: - Add new config file_cache_warmup_download_rate_limit_bytes_per_second (default 100MB/s, ≤0 means no limit) - Introduce DEFINE_ON_UPDATE macro to register callbacks triggered when config values change at runtime - Implement S3RateLimiterHolder in ExecEnv for node-level warmup download rate limiting - Add bvar metrics for monitoring: warmup_download_rate_limit_latency, warmup_download_rate_limit_ns, warmup_download_rate_limit_exceed_req_num - Rate limit can be dynamically updated at runtime without restart Problem & Solution When file cache warmup tasks download large amounts of data from remote storage (e.g., S3), they can consume excessive bandwidth and impact normal query performance. This PR introduces a configurable rate limiter specifically for warmup downloads to prevent bandwidth saturation. The DEFINE_ON_UPDATE mechanism enables runtime config changes to take effect immediately by triggering registered callbacks, allowing the rate limiter to be adjusted dynamically without service restart. Test plan - Added unit tests for DEFINE_ON_UPDATE callback mechanism (config_on_update_test.cpp) - Tests cover Int64, String, and Bool config types with callback invocation verification
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pick #60180