1.2.0-rc0

What's Changed

• refactor(log): introduce async close by @DaniilRoman in #1503
• feat(s3stream): Modify the memory usage detection interval in ByteBufAlloc through environment variables. by @llzcx in #1557
• fix(s3stream): fix npe when controller not config wal path by @lifepuzzlefun in #1560
• feat(s3stream): add random jitter delay time for StreamObjectsCompaction by @lifepuzzlefun in #1561
• chore: adjust readme file by @KaimingWan in #1549
• feat(e2e): add timeout for ConsumerPerformanceService. by @llzcx in #1563
• feat(e2e): kafka.py JVM OPT uses parameter passing instead of fixed values. by @llzcx in #1562
• test(core): fix broken test by @SCNieh in #1565
• feat(e2e): Memory occupancy E2E test. by @llzcx in #1564
• refactor(s3stream): rename newMultiPartWriter to newLargeObjectWriter and change visibility to public by @warr99 in #1559
• feat: object storage enhancement by @Chillax-0v0 in #1570
• refactor(s3stream): change NetworkBandwidthLimiter attribute access modifiers by @warr99 in #1569
• feat: Downgrade argparse4j to 0.7.0 by @Amitbhave in #1566
• feat(metadata): remove useless s3object fields by @superhx in #1571
• feat(metadata): remove useless fields from s3streamobject by @superhx in #1573
• feat(s3stream): introduce fast retry mechanism in ObjectStorage by @ShadowySpirits in #1574
• feat(s3stream): batch delete in AbstractObjectStorage by @lifepuzzlefun in #1572
• feat(metadata): use delta list for so & sso by @superhx in #1575
• feat(metadata): replace deltamap to timelinehashmap by @superhx in #1580
• fix: use Properties#store to save properties by @Chillax-0v0 in #1581
• fix(test): fix small timeout cause testHighTrafficBatchDelete fail by @lifepuzzlefun in #1583
• chore: change the copyright to AutoMQ HK Limited by @daniel-y in #1586
• feat(core): optimize indexing performance by using sparse index cache by @SCNieh in #1585
• feat(metadata): clean up sparse index map when empty by @SCNieh in #1587
• feat(s3stream): build read options when recover s3 wal by @ShadowySpirits in #1589
• feat(metadata): add sso in-memory index cache by @SCNieh in #1588
• feat(s3stream): fix buffer leak by @ShadowySpirits in #1591
• feat(metadata): add version control to sparse index serialization by @SCNieh in #1590
• feat(wal): graceful shutdown optimization #1578 by @CLFutureX in #1579
• fix(s3stream): add temp method to get bucketId by @SCNieh in #1594
• feat(s3stream): limit s3 wal object size by @ShadowySpirits in #1593
• refactor(s3stream): remove minPartSize parameter from newLargeObjectWriter method by @warr99 in #1598
• test(s3stream): add waiting for InflightReadaheadCf by @llzcx in #1596
• feat(s3stream): refactor s3 client by @ShadowySpirits in #1600
• fix(perf): fix options in "automq-perf-test.sh" by @Chillax-0v0 in #1605
• feat(s3stream): support config MAJOR_V1 skip compact small object by @lifepuzzlefun in #1584
• fix(log): fix consumer offsets load missing (#1606) by @superhx in #1607
• feat(metadata): optimize object lifecycle check by @superhx in #1603
• fix(metadata): fix buffer leak by @SCNieh in #1609
• feat(s3stream): change LocalStreamRangeIndexCache to normal instance by @SCNieh in #1597
• chore: use gif and update badges by @KaimingWan in #1592
• fix(auto_balancer): fix potential memory leak by @SCNieh in #1612
• feat(issues1608): clean up useless range metadata by @superhx in #1610
• fix(metadata): remove duplicated buffer release by @SCNieh in #1614
• fix(s3stream): metric record zero value when read composite index by @lifepuzzlefun in #1601
• feat(s3stream): extract uri util methods by @SCNieh in #1615
• fix(s3stream): parse with raw query string to prevent double decode by @SCNieh in #1617
• fix(s3stream): fix get the v0 automq version when recover by @superhx in #1618
• feat(api): add automq node api by @superhx in #1616
• fix(s3stream): release buffer on exception by @SCNieh in #1622
• fix(s3stream): add compaction delay metrics by @SCNieh in #1623
• feat(e2e): jmx.py supports obtaining the maximum average value of all windows by @llzcx in #1595
• test(e2e): add quota test by @llzcx in #1604
• feat(controller): add node manager by @superhx in #1627
• fix(issue1620): fix flaky test by @SCNieh in #1621
• feat(metadata): use lru cache to control the overall size of node sparse index cache by @SCNieh in #1599
• fix(s3stream): fix version get npe by change the s3stream load timing by @superhx in #1632
• feat(core): optimize LoadAwarePartitionLeaderSelector by @SCNieh in #1633
• fix: update docker-release.yml to fetch tags before determining… by @Gezi-lzq in #1631
• feat(e2e): JMX tool supports obtaining metrics for each broker by @llzcx in #1626
• fix(s3stream): fix index out of bound exception by @SCNieh in #1636
• feat(core): add quota metrics into jmx metrics mapping by @ShadowySpirits in #1637
• fix(api): fix node metadata by @superhx in #1638
• fix(s3stream): fix prepared object deletion by @superhx in #1639
• feat(s3stream): report delta mean values for selected jmx metrics by @SCNieh in #1629
• feat(metadata): add sanity check of getObjects result by @SCNieh in #1634
• feat(core): refactor request quota limitation by @ShadowySpirits in #1635
• feat(admin): add get nodes admin by @superhx in #1641
• test(s3stream): disable unstable test by @SCNieh in #1640
• fix(issues1642): fix bucket not match by @superhx in #1643
• fix(s3stream): release byteBuf before complete readTask by @warr99 in #1644
• fix(ab): fix partition status update by @SCNieh in #1647
• feat(s3stream): add new wal config by @superhx in #1646
• feat(s3stream): IdURI support path substitude by @superhx in #1651
• chore: get extensions in IdURI by @Chillax-0v0 in #1654
• fix(e2e): add init.py to ensure proper module import by @llzcx in #1645
• test(e2e): add autobalancer test by @llzcx in #1653
• feat(s3stream): id uri encode by @superhx in #1656
• fix(metadata): avoid throwing exception to crash process by @SCNieh in #1649
• docs: Correct spelling error in README.md by @jasminaaa20 in #1652
• chore(s3stream): make failover more extensible by @superhx in #1658
• feat(s3stream): add configurable retry delay for objectStorage by @warr99 in #1659
• feat(metrics): use uri to simplify metrics configuration by @SCNieh in #1657
• feat: convert IdURI to BucketURI by @Chillax-0v0 in #1660
• perf(s3stream): increase the object overhead of StreamRecordBatch by @Chillax-0v0 in #1661
• refactor(s3stream): move component initialization to start by @superhx in #1664
• fix(s3stream): fix startup NPE by @superhx in #1667
• feat(s3stream): support burst upload for upload WAL task to reduce failover time. by @lifepuzzlefun in #1662
• feat(metrics): optimize s3 operation metrics interface by @SCNieh in #1673
• fix(metrics): fix histogram duplicated register by @superhx in #1672
• fix(metrics): prevent potential NPE by @SCNieh in #1674
• feat(metrics): upgrade OpenTelemetry dependencies versions by @SCNieh in #1665
• feat(e2e): add main_kos_test_suite5 and nightly job by @llzcx in #1655
• fix(metrics): fix duplicated operation name by @SCNieh in #1675
• refactor(s3stream/wal): allow reset without recover by @Chillax-0v0 in #1676
• chore: expose streamControlManager in QuorumController by @Chillax-0v0 in #1677
• chore: expose nodeControlManager and featureControlManager by @Chillax-0v0 in #1679
• fix(e2e): increase poll timeout to prevent test fail by @SCNieh in #1680
• fix(e2e): rename method name that is not a test by @SCNieh in #1681
• perf(core): batch persistent meta when delete large scale segment by @lifepuzzlefun in #1670
• feat(license): optimize the license, make it clearer for developer by @daniel-y in #1682
• refactor: data clumps by @compf in #1663
• feat(s3Stream): fix excessive cache release in concurrent scenarios. by @CLFutureX in #1685
• fix(e2e): ignore test that's not compatible with AutoMQ by @SCNieh in #1684
• feat(issues1446): avoid accidentally fence brokers when controller overload by @superhx in #1686
• feat(s3stream): optimize lock for s3 wal by @ShadowySpirits in #1688
• feat(metrics): enable ops exporter by default by @SCNieh in #1690
• feat(auto_balancer): enable AutoBalancer by default by @SCNieh in #1687
• fix(issues1689): retain image before use by @superhx in #1692
• feat: get all nodes' metadata by @Chillax-0v0 in #1691
• fix(core): Fix integer overflow in bytesNeed calculation by @ArickNan in #1694
• feat(controller): make broker nodeId starts from 1000 by @superhx in #1696
• fix(s3stream): open RecordAccumulator#unsafeUpload for testing by @ShadowySpirits in #1695
• refactor: expose methods of NodeRuntimeInfoGetter in NodeControlManager by @Chillax-0v0 in #1698
• feat(s3stream): fix maven dependency for s3stream e2e by @ShadowySpirits in #1700
• feat(telemetry): serialize the metrics label to prometheus format by @ShadowySpirits in #1702
• feat(cli): remove old cluster bootstrap by @superhx in #1703
• feat(s3stream): stop the timer wheel when shutdown s3 storage by @ShadowySpirits in #1704
• feat(metrics): remove unnecessary resource attributes by @SCNieh in #1697
• feat(controller): add has opening streams method by @superhx in #1707
• chore(cli): rename configs to config by @superhx in #1708
• chore(logger): suppress object not exist error by @superhx in #1709
• feat(s3stream): isolate the wal object from previous epoch by @ShadowySpirits in #1699
• feat(s3stream): test s3 wal in s3stream e2e by @ShadowySpirits in #1705
• feat(tests): test s3 wal in automq enhanced e2e tests by @ShadowySpirits in #1712
• feat(tests): implement append method properly for s3 wal by @ShadowySpirits in #1711
• feat(s3stream): force close streams when WAL is failed by @superhx in #1713
• feat(s3stream): extract real exception by @CLFutureX in #1710
• fix(s3stream): fix buffer release by @ShadowySpirits in #1716
• feat(e2e): add auto-balancer and s3-object logs by @llzcx in #1701
• fix(log): fix async close not wait by @superhx in #1724
• fix(controller): npe when cleanup scale in node by @lifepuzzlefun in #1728
• feat(core): expose more java modules by @lifepuzzlefun in #1732
• feat(bin): optimise kafka-run-class.sh by @ShadowySpirits in #1733
• feat(s3stream): handle IOException thrown during recovering data from the WAL by @Chillax-0v0 in #1720
• feat(metrics): support customize base labels by @SCNieh in #1735
• docs(bin): optimise kafka-run-class.sh by @Chillax-0v0 in #1736
• feat(test): only run s3stream e2e after merging pr by @ShadowySpirits in #1734
• fix(issues1737): log close await append callback complete by @superhx in #1738
• fix(s3stream): halt when the node is fenced by @superhx in #1739
• fix(issues1740): fix log close await deadlock by @superhx in #1741
• fix(s3stream): correct delete object path to use multiPartPath by @warr99 in #1742
• feat(image): add image get objects endless loop detect by @superhx in #1744
• fix(s3stream): make ObjectWriter#write synchronized by @Chillax-0v0 in #1745
• test(e2e): add s3 leakage test by @llzcx in #1719
• fix(ElasticLogManager): terminate the unending shutdown by completing INIT_FUTURE by @Chillax-0v0 in #1750
• fix(config): add default metrics reporter only in broker node by @SCNieh in #1751
• feat(auto_balancer): optimize broker schedule threshold by @SCNieh in #1747
• feat(cli): describe cluster by @superhx in #1754
• refactor(s3stream): change access permission by @warr99 in #1755
• feat(metadata): unregister nodes by @Chillax-0v0 in #1756
• fix(s3stream): support idempotent delete composite object by @superhx in #1757
• refactor(s3stream): make maxBatchSize in DeleteObjectsAccumulator configurable by @warr99 in #1758
• feat(s3stream): remove default read options by @superhx in #1759
• feat(auto_balancer): preserve out-of-sync broker and partition in cluster model snapshot by @SCNieh in #1760
• feat(metadata): reject requests to unregister a node with already open streams by @Chillax-0v0 in #1761
• fix(metadata): refine error handler for cf by @SCNieh in #1763
• feat(metadata): catch loadStreamSetObjectInfo unexpected exception by @superhx in #1765
• chore(logger): log composite object load fail by @superhx in #1766
• feat(s3stream): optimize the error log by @ShadowySpirits in #1768
• feat(idempotent): update check sequence timeout to 40s by @superhx in #1770
• feat(s3stream): add strictBathLimit option for s3 wal by @ShadowySpirits in #1769
• fix(readme): adjust the heading level by @ShadowySpirits in #1775
• feat(auto_balancer): allow change broker rack on register (#1771) by @SCNieh in #1776
• fix(metadataimage): fix getObjects not complete cf when throw exception by @lifepuzzlefun in #1764
• feat(s3stream): change s3 wal object path by @ShadowySpirits in #1777
• refactor(s3stream): refactor batch delete objects by @warr99 in #1767
• fix(s3stream): fix exception handling in s3 wal by @ShadowySpirits in #1773
• chore(s3stream/metadata): more logs during pending fetch by @Chillax-0v0 in #1780
• fix(issues1774): fix async lru cache ConcurrentModificationException by @superhx in #1781
• test(e2e): add compaction test. by @llzcx in #1772
• feat(s3stream): optimize the execution order of the release method for ByteBuf by @CLFutureX in #1779
• feat(auto_balancer): assign partitions within rack on broker fence (#1778) by @SCNieh in #1783
• fix(issues1784): fix upload burst cause object index out of order by @superhx in #1785
• fix(s3stream): fix AsyncLRUCache to support remove by @superhx in #1786
• fix(metadata): add object header size to cache size estimation by @SCNieh in #1791
• feat(s3stream): add thread prefix in objectStorage by @lifepuzzlefun in #1789
• perf: use netty buffer to pool memory by @Chillax-0v0 in #1788
• feat(s3stream): add missing getter method by @SCNieh in #1793
• chore(logger): move controller commit sso log to broker by @superhx in #1794
• feat(metrics): add topic partition distribution metric (#1782) by @SCNieh in #1792
• feat: add introduction link by @KaimingWan in #1795
• fix(core): Revert "perf: use netty buffer to pool memory (#1788)" by @ShadowySpirits in #1797
• perf: use a new GrowableMultiBufferSupplier to avoid memory waste by @Chillax-0v0 in #1800
• feat(log): add error level log in logSuppressor by @warr99 in #1799
• fix(issues1798): fix stream reader memory leak by @superhx in #1801
• fix(metrics): topic partition count metrics should be reported by all brokers by @SCNieh in #1802
• feat(s3stream): optimize the execution sequence to avoid deep depende… by @CLFutureX in #1790
• fix(auto_balancer): fix load balancer threshold by @SCNieh in #1805
• fix(s3stream): fix dead lock issue by @ShadowySpirits in #1810
• feat(all): merge apache kafka 3.8.0 771b957 by @superhx in #1813
• fix(metadata): fix sparse index serialization capacity by @SCNieh in #1815
• fix(e2e): adaptation to pr: #1794 by @llzcx in #1811
• fix(cli): add cli utils back by @superhx in #1816
• chore(logger): move trim log to broker by @superhx in #1820
• chore(config): change quorum default timeout to 5s by @superhx in #1824
• chore(workflows): e2e support release runner by @superhx in #1826
• fix(s3strea/wal): handle IOException during flushing WAL header by @Chillax-0v0 in #1828
• fix(test): cherry-pick #1830 #1835 by @superhx in #1836
• feat(s3stream): add AsyncLRUCache metric (#1829) by @SCNieh in #1837
• fix(s3stream): fix storage failure handle deadlock by @superhx in #1839
• fix(s3stream/wal): fail all IO operations once the WAL is failed by @Chillax-0v0 in #1841
• feat(metadata): limit the size of sparse index cache by @SCNieh in #1833
• feat(issues1842): cleanup metastream kv after deleting topic by @superhx in #1844
• fix(s3stream): error callback by @superhx in #1846
• fix(revert): error callback by @superhx in #1849
• feat(metadata): avoid serialize empty index by @SCNieh in #1853
• feat(metadata): limit sparse index cache update interval by @SCNieh in #1850
• feat(s3stream): support call #close(force=true) after #close(force=false) by @superhx in #1855
• feat(metadata): prune invalid sparse index with image on startup by @SCNieh in #1857
• feat(s3stream): add network rate limiter for s3 wal to export metrics by @ShadowySpirits in #1861
• feat(e2e): use cloud vms for e2e tests; move wal path by @mooc9988 in #1864
• fix(e2e): always archive result artifacts by @mooc9988 in #1867
• fix(storage): regard all produce requests as duplicate whose sequence… by @Chillax-0v0 in #1868
• revert: enable LocalStreamRangeIndexCacheTest#testEvict by @Chillax-0v0 in #1870
• fix(metadata): catch exception to prevent unnecessary error log by @SCNieh in #1873
• perf(s3stream): check the logger level before trace and debug by @Chillax-0v0 in #1876
• chore(tools/perf): increase the message sending rate during the warmup to accelerate JVM warmup by @Chillax-0v0 in #1881
• perf(s3stream/wal): reuse the ByteBuf for record headers (#1877) by @Chillax-0v0 in #1878
• fix(test): fix consumer mode e2e by increase timeout by @superhx in #1886
• fix(e2e): fix bug by @llzcx in #1887
• chore(logger): change e2e streams log level to INFO by @superhx in #1890
• perf(s3stream): optimize the critical section in s3 wal (#1880) by @ShadowySpirits in #1893
• fix(s3stream/wal): fix memory leak by @Chillax-0v0 in #1897
• fix(e2e): fix data map format by @mooc9988 in #1899
• fix(metadata): fix sparse index leak on compaction split by @SCNieh in #1895
• fix(metadata): fix evict sparse index cache potential endless loop by @SCNieh in #1901
• feat(metadata): add metrics for sparse index cache monitor by @SCNieh in #1903
• fix(metadata): prevent concurrent uploading of local sparse index cache by @SCNieh in #1905
• fix(metadata): fix wrong stream number in serialization by @SCNieh in #1907
• feat(s3stream): support change checksum algorithm by @ShadowySpirits in #1909
• feat(s3stream): add docs for checksumAlgorithm configuration by @ShadowySpirits in #1910
• fix(test): set dev version to 3.8.0 by @superhx in #1911
• fix(metadata): fix interrupted batch upload by @SCNieh in #1912
• perf(config): increase the default index interval from 4KiB to 1MiB by @Chillax-0v0 in #1914
• feat(s3stream): unify object not exist exception by @superhx in #1921
• fix(e2e): fix bug by @llzcx in #1923
• feat(s3stream): limit the inflight fast retry request count by @ShadowySpirits in #1924
• chore(test): set runner to e2e by @superhx in #1928
• feat(metadata): expire node sparse index cache after write by @SCNieh in #1926
• feat(auto_balancer): limit the max number of nodes to reassign partit… by @SCNieh in #1934
• feat(metadata): blocking wait for uploading local index cache on stre… by @SCNieh in #1932
• feat(s3stream): ensure compaction could be executed by @superhx in #1935
• feat(s3stream): add checksum for s3 wal by @ShadowySpirits in #1940
• docs(s3stream/version): statement of features supported by S3Stream V1 by @Chillax-0v0 in #1942
• fix(s3stream): resolve unintended buffer reuse by @ShadowySpirits in #1946
• fix(s3stream): fix buffer leak by @ShadowySpirits in #1949
• fix(s3stream): change reject handler to prevent task lost by @SCNieh in #1957
• fix(issues1960): allow UNSET attributes for version 1.1 by @superhx in #1961
• fix(log): use the same view to calculate trim offsets by @Chillax-0v0 in #1972
• fix(s3stream): fix namespace by @ShadowySpirits in #1978
• feat(core): default configuration optimization by @SCNieh in #1980
• fix(auto_balancer): fix string format by @SCNieh in #1988
• feat(auto_balancer): limit the maximum time to execute actions by @SCNieh in #1990
• fix(auto_balancer): fix broker status change by @SCNieh in #1992
• chore: beautify ElasticLogMeta in logs by @Chillax-0v0 in #1996
• chore(util): implement IdURI#toString by @Chillax-0v0 in #1997
• fix(issues1999): fix Processor.channelContexts memory leak by @superhx in #2002
• feat(version): bump to 1.2.0-rc0 by @superhx in #2003
• fix(issue2004): fix AutoBalancerMetricsReporter cannot process with t… by @SCNieh in #2006
• fix(core): release all records before delayed fetch by @ShadowySpirits in #2009
• perf(DelayedFetch): only try to fast read on complete a delayed fetch by @Chillax-0v0 in #2013

New Contributors

• @DaniilRoman made their first contribution in #1503
• @Amitbhave made their first contribution in #1566
• @jasminaaa20 made their first contribution in #1652
• @compf made their first contribution in #1663
• @ArickNan made their first contribution in #1694

Full Changelog: 1.2.0-beta0...1.2.0-rc0