[AMORO-4037] Add support for dynamic refresh interval for table metadata refreshing in TableRuntimeRefreshExecutor by zhangwl9 · Pull Request #4038 · apache/amoro

zhangwl9 · 2026-01-14T06:46:41Z

Why are the changes needed?

Close #4037.

Brief change log

The new feature involves two methods in "TableRuntimeRefreshExecutor".
- #getNextExecutingTime()
  - When dynamic refresh interval is enabled，retrieving the latest refresh interval from "DefaultTableRuntime" to determine the next execution time for "TableRuntimeRefreshExecutor#execute"
- #execute()
  - Depending on the evaluated results, determine whether the table requires optimization.
  - When the evaluation table requires no optimization, extend the interval for its next refresh (incrementally increasing the interval by a fixed step).
  - When evaluating table requiring optimization, shorten its next refresh interval (halving the interval each time).
dynamic refresh interval in [ refresh-tables.interval(1 min)，Long.MAX_VALUE]

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

zhangwl9 · 2026-01-16T01:59:22Z

@zhoujinsong @turboFei @xxubai Could you help me review it when you are free? Thanks!

Copilot

Pull request overview

This PR implements a dynamic refresh interval feature for table metadata refreshing in TableRuntimeRefreshExecutor. The feature uses an AIMD (Additive Increase Multiplicative Decrease) algorithm to adjust refresh intervals based on whether tables need optimization, helping to reduce resource consumption for healthy tables while maintaining responsiveness for tables requiring optimization.

Changes:

Added configuration properties for adaptive refresh intervals with configurable maximum interval and increase step
Implemented adaptive interval calculation logic that increases intervals for healthy tables and decreases them for tables needing optimization
Added comprehensive test coverage for the adaptive interval functionality

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/user-guides/configurations.md	Added documentation for three new configuration properties related to adaptive refresh intervals
amoro-format-iceberg/src/main/java/org/apache/amoro/table/TableProperties.java	Defined constants for adaptive refresh table properties and their default values
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java	Added fields, getters, setters, and updated equals/hashCode/toString methods for adaptive refresh configuration
amoro-ams/src/main/java/org/apache/amoro/server/table/TableConfigurations.java	Integrated parsing of adaptive refresh properties from table properties into OptimizingConfig
amoro-ams/src/main/java/org/apache/amoro/server/table/DefaultTableRuntime.java	Added fields and methods to track latest refresh interval and optimization evaluation status
amoro-ams/src/main/java/org/apache/amoro/server/scheduler/inline/TableRuntimeRefreshExecutor.java	Implemented core adaptive interval logic with AIMD algorithm and updated execute method to apply adaptive intervals
amoro-ams/src/test/java/org/apache/amoro/server/scheduler/inline/TestTableRuntimeRefreshExecutor.java	Added comprehensive test cases covering various adaptive interval scenarios including boundary conditions

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/user-guides/configurations.md

klion26

Thanks for the contribution. I left some comments; please let me know what you think. thanks.

IMHO, this feature is valuable(especially when users use HMS as the catalog), as we can't use the event-based trigger as RestCatalog, but when there are many "static table" in the catalog, it will be a great burden for AMS to evaluate these tables, and changing the interval for every table is not a doable(as there are so many tables), and keep the default behavior as the same as before, would not add any inconvenience if we don't need it.

klion26 · 2026-01-27T11:29:28Z

docs/user-guides/configurations.md

+| self-optimizing.min-plan-interval                    | 60000            | The minimum time interval between two self-optimizing planning action                                                                                                                                                               |
+| self-optimizing.filter                               | NULL             | Filter conditions for self-optimizing, using SQL conditional expressions, without supporting any functions. For the timestamp column condition, the ISO date-time formatter must be used. For example: op_time > '2007-12-03T10:15:30'. |
+| self-optimizing.refresh-table.adaptive.enabled       | false            | Whether to enable adaptive refresh interval for refreshing table metadata                                                                                                                                                           |
+| self-optimizing.refresh-table.adaptive.max-interval  | 3600000(1 hour)  | The maximum time interval in milliseconds to refresh table metadata                                                                                                                                                                 |


Maybe we can set the max-interval to an unset value, so that we can keep the behavior same as before the current pr.

klion26 · 2026-01-27T11:31:06Z

amoro-format-iceberg/src/main/java/org/apache/amoro/table/TableProperties.java

  public static final int SNAPSHOT_MIN_COUNT_DEFAULT = 1;

+  public static final String SELF_OPTIMIZING_REFRESH_TABLE_ADAPTIVE_ENABLED =
+      "self-optimizing.refresh-table.adaptive.enabled";


If we need to set self-optimizing.refresh-table.adaptive.enabled and self-optimizing.refresh-table.adaptive.max-interval, how about using only self-optimizing.refresh-table.adaptive.max-interval? Some value (like -1 or zero) means that disable it, and some other value means that to enable it.

klion26 · 2026-01-27T11:31:41Z

amoro-format-iceberg/src/main/java/org/apache/amoro/table/TableProperties.java

+  public static final boolean SELF_OPTIMIZING_REFRESH_TABLE_ADAPTIVE_ENABLED_DEFAULT = false;
+  public static final String SELF_OPTIMIZING_REFRESH_TABLE_ADAPTIVE_MAX_INTERVAL =
+      "self-optimizing.refresh-table.adaptive.max-interval";
+  public static final long SELF_OPTIMIZING_REFRESH_TABLE_ADAPTIVE_MAX_INTERVAL_DEFAULT =


How about add the time unit in the name, SELF_OPTIMIZING_REFRESH_TABLE_ADAPTIVE_MAX_INTERVAL_DEFAULT -> SELF_OPTIMIZING_REFRESH_TABLE_ADAPTIVE_MAX_INTERVAL_MS_DEFAULT

Same as the other places

klion26 · 2026-01-28T02:56:31Z

...-ams/src/main/java/org/apache/amoro/server/scheduler/inline/TableRuntimeRefreshExecutor.java

      if (optimizingConfig.isMetadataBasedTriggerEnabled()
          && !MetadataBasedEvaluationEvent.isEvaluatingNecessary(
              optimizingConfig, table, tableRuntime.getLastPlanTime())) {
+        tableRuntime.setLatestEvaluatedNeedOptimizing(false);


Is it better that we change the return value of tryEvaluatingPendingInput to boolean -- indicate whether we need to optimize after evaluating?
So that we can call tableRuntime.setLatestEvaluteNeedOptimizing in one place in execute below,

Please let me what do you think about it.

...-ams/src/main/java/org/apache/amoro/server/scheduler/inline/TableRuntimeRefreshExecutor.java

klion26 · 2026-01-28T03:01:39Z

.../src/test/java/org/apache/amoro/server/scheduler/inline/TestTableRuntimeRefreshExecutor.java

+    executor.execute(mockTableRuntime);
+
+    // Verify that setLatestRefreshInterval was called with the expected value
+    verify(mockTableRuntime, Mockito.times(1)).setLatestRefreshInterval(expectedInterval);


Instead using verify here to validate, is there any way we can validate the value of the true interval after executor.execute in L147.

The true interval is the value keeped in the table runtime.

klion26 · 2026-01-28T03:05:07Z

.../src/test/java/org/apache/amoro/server/scheduler/inline/TestTableRuntimeRefreshExecutor.java

+
+    // Test unhealthy table (need optimizing) - interval should decrease
+    mockTableRuntime =
+        getMockTableRuntimeWithAdaptiveInterval(tableRuntime, true, true, INTERVAL * 2);


Is there a way to reuse this tableRuntime so we can observe the change in interval before and after two consecutive execute calls? Reusing a single tableRuntime function would also be more suitable for production scenarios.

klion26

Thanks for the update, I've left some comments, please take another look.

klion26 · 2026-01-30T09:17:07Z

...-ams/src/main/java/org/apache/amoro/server/scheduler/inline/TableRuntimeRefreshExecutor.java

+          tableRuntime.getTableIdentifier());
+      return false;
+    } else {
+      logger.debug(


Could we add some comments to describle the "semantics" of the return value, and why would this case would be true?

From below, the name is needOptimizing, looks like whethere we need to do an opitmizing, but in this branch, we don't need to optimizing in current loop? as there is an ongoing optimizing process

klion26 · 2026-01-30T09:19:34Z

...-ams/src/main/java/org/apache/amoro/server/scheduler/inline/TableRuntimeRefreshExecutor.java

+      }
+
+      // Update adaptive interval according to evaluated result.
+      if (defaultTableRuntime.getOptimizingConfig().getRefreshTableAdaptiveMaxIntervalMs() > 0) {


Maybe we can add a function in OptimizingConfig which return true/false show whethere we enabled this feature

klion26 · 2026-01-30T09:31:40Z

docs/user-guides/configurations.md

+| self-optimizing.full.rewrite-all-files                  | true             | Whether full optimizing rewrites all files or skips files that do not need to be optimized                                                                                                                                              |
+| self-optimizing.min-plan-interval                       | 60000            | The minimum time interval between two self-optimizing planning action                                                                                                                                                                   |
+| self-optimizing.filter                                  | NULL             | Filter conditions for self-optimizing, using SQL conditional expressions, without supporting any functions. For the timestamp column condition, the ISO date-time formatter must be used. For example: op_time > '2007-12-03T10:15:30'. |
+| self-optimizing.refresh-table.adaptive.max-interval-ms  | -1               | The maximum time interval in milliseconds to refresh table metadata. The default value is -1, which disables the adaptive refresh.                                                                                                                |


Maybe we need to add some description here: after this has been enabled, the interval may be larger than defaultTableRuntime.getOptimizingConfig().getMinorLeastInterval() * 4L / 5

klion26 · 2026-01-30T09:41:02Z

.../src/test/java/org/apache/amoro/server/scheduler/inline/TestTableRuntimeRefreshExecutor.java

+    createDatabase();
+    createTable();
+
+    DefaultTableRuntime tableRuntime =


Could you please reuse the same table runtime, in the current test, we'll create every new table runtime in buildTableRuntimeWithAdaptiveRefresh

If we need to update the config for an existing table runtime, maybe we can add a new class who extend DefaultTableRuntime and overrides the getStore function.

class TestTableRuntime extends DefaultTableRuntime { TableRuntimeStore testStore; TestTableRuntime(TableRuntimeStore store) { ... this.testStore = store } void updateRuntimeStore(TableRuntimeStore newStore) { ... this.testStore = newStore; } @Override TableConfiguration getStore() { this.testStore; }

klion26 · 2026-02-02T02:18:09Z

.../src/test/java/org/apache/amoro/server/scheduler/inline/TestTableRuntimeRefreshExecutor.java

+    // Test minimum boundary - interval should not below INTERVAL
+    // The unhealthy table (need optimizing) - interval should decrease half
+    // The currentInterval is INTERVAL + STEP, the latest interval is INTERVAL not (INTERVAL + STEP)
+    // / 2


Place (INTERVAL + STEP) / 2 in the same line is better than 2 lines

codecov-commenter · 2026-02-03T08:54:04Z

Codecov Report

❌ Patch coverage is 61.79775% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.68%. Comparing base (5427715) to head (fbbd84f).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...java/org/apache/amoro/config/OptimizingConfig.java	0.00%	18 Missing ⚠️
.../scheduler/inline/TableRuntimeRefreshExecutor.java	72.88%	12 Missing and 4 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #4038      +/-   ##
============================================
- Coverage     28.70%   28.68%   -0.02%     
- Complexity     3944     3951       +7     
============================================
  Files           656      654       -2     
  Lines         52382    52385       +3     
  Branches       6642     6630      -12     
============================================
- Hits          15035    15026       -9     
- Misses        36249    36265      +16     
+ Partials       1098     1094       -4

Flag	Coverage Δ
core	`28.68% <61.79%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

klion26 · 2026-02-04T11:03:41Z

amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java

    return this;
  }

+  public boolean isRefreshTableAdaptiveEnabled(long minInterval) {


Why do we need to pass in this parameter? will compare the refreshTableAdaptiveMaxIntervalMs to a default value in the current class be better?

The logic here seems is different than the doc below The maximum time interval in milliseconds to refresh table metadata. 0 means disable adaptive refresh.?

…g in the TableRuntimeRefreshExecutor.

…e.max-interval-ms

github-actions bot added type:docs Improvements or additions to documentation module:ams-server Ams server module module:common labels Jan 14, 2026

zhangwl9 changed the title ~~[AMORO-4037] Add support for dynamic refresh interval for table metadata refreshing~~ [AMORO-4037] Add support for dynamic refresh interval for table metadata refreshing in TableRuntimeRefreshExecutor Jan 14, 2026

zhangwl9 force-pushed the AMORO-dynamic-refresh-interval-dev branch from 60e829e to 298152d Compare January 15, 2026 07:06

zhangwl9 force-pushed the AMORO-dynamic-refresh-interval-dev branch from 298152d to 4fec76a Compare January 21, 2026 05:42

turboFei requested a review from Copilot January 22, 2026 04:46

Copilot AI reviewed Jan 22, 2026

View reviewed changes

docs/user-guides/configurations.md Outdated Show resolved Hide resolved

zhangwl9 force-pushed the AMORO-dynamic-refresh-interval-dev branch 2 times, most recently from 16b59c2 to 3298bcf Compare January 27, 2026 06:15

klion26 reviewed Jan 28, 2026

View reviewed changes

zhangwl9 force-pushed the AMORO-dynamic-refresh-interval-dev branch 3 times, most recently from 1a658f8 to 4f0ca86 Compare January 29, 2026 09:25

klion26 force-pushed the AMORO-dynamic-refresh-interval-dev branch from 4f0ca86 to 72aad3b Compare January 30, 2026 08:23

klion26 reviewed Feb 2, 2026

View reviewed changes

zhangwl9 force-pushed the AMORO-dynamic-refresh-interval-dev branch 3 times, most recently from 808f66b to fbbd84f Compare February 3, 2026 08:18

zhangwl9 force-pushed the AMORO-dynamic-refresh-interval-dev branch from fbbd84f to 5955420 Compare February 3, 2026 09:02

klion26 reviewed Feb 4, 2026

View reviewed changes

zhangwl9 force-pushed the AMORO-dynamic-refresh-interval-dev branch 2 times, most recently from 1e01c3f to 08871a7 Compare February 6, 2026 07:58

张文领 added 5 commits February 6, 2026 16:02

Add support for dynamic refresh interval for table metadata refreshin…

222e4bc

…g in the TableRuntimeRefreshExecutor.

fixup

f17c152

fixup

b9877f1

The validation parameter maxInterval must be greater than minInterval.

37d8369

fixup the invalid configuration self-optimizing.refresh-table.adaptiv…

5d564df

…e.max-interval-ms

zhangwl9 force-pushed the AMORO-dynamic-refresh-interval-dev branch from 08871a7 to 5d564df Compare February 6, 2026 08:02

Conversation

zhangwl9 commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

Brief change log

How was this patch tested?

Documentation

Uh oh!

zhangwl9 commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

klion26 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

klion26 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Feb 3, 2026

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zhangwl9 commented Jan 14, 2026 •

edited

Loading

zhangwl9 commented Jan 16, 2026 •

edited

Loading