v1.1.0

Restate 1.1.0

IMPORTANT for SDK-Typescript users: Only when upgrading from 1.0.x to 1.1.x you MUST rediscover all the existing deployments using restate dp register <address> --force. You don't need to update the SDK, nor change the code.

Notable changes:

• Restate now sets the x-restate-invocation-id header when making service invocations to support load balancers implementing "sticky" routing
• Improved forward- and backward-compatibility checks with data store on service startup
• When discovering https service deployments, now both HTTP/1.1 and HTTP/2 are supported, with negotiation performed using ALPN. When discovering http deployments, HTTP/2 prior knowledge is assumed, but you can override it to use HTTP/1.1 by passing the --use-http1.1 flag to restate deployments register.
• We now publish a Docker image restatedev/restate-cli containing the restate CLI tool (#1758)
• Restate CLI and SQL interface now contain several additional fields to improve debugging your service invocations.

Notable bug fixes:

• Service-to-service invocations will now correctly propagate headers
• Tracing context header injection at the Restate ingress endpoint works as expected

Infrastructure updates:

• Forward compatibility with the upcoming Service Invocation Protocol V2, enabling granular retry policies
• Introduces the restatectl administrative tool which helps operators manage Restate cluster configuration
• Foundational work on the Restate distributed log and data partitioning for upcoming distributed deployment support

What's Changed

• Fix pinned_deployment_id is printed incorrectly in SQL by @slinkydeveloper in #1612
• [RocksDb] Fixing RocksDbPerfGuard's semantics and performance by @AhmedSoliman in #1606
• Use GET for service discovery by @slinkydeveloper in #1613
• Fix panics datafusion by @slinkydeveloper in #1618
• Update compatibility map to include 1.0.0 as a new breaking version by @tillrohrmann in #1616
• Check for backward compatibility when validating the cluster marker by @tillrohrmann in #1619
• [config] Adds invoker segment queue limit to configuration file by @AhmedSoliman in #1622
• Unify mocks and test-util feature in crates by @tillrohrmann in #1627
• Use different directories for cluster marker tests by @tillrohrmann in #1628
• [CLI] Bootstrapping restatectl admin command line by @AhmedSoliman in #1625
• Lazily calculate value when using StyledTable::add_kv_row_if and predicate is true by @tillrohrmann in #1632
• feat: update pgwire to latest release by @sunng87 in #1634
• Generate documentation of SQL tables by @slinkydeveloper in #1623
• Delete the node dir when wiping all by @tillrohrmann in #1630
• Let shuffle always use get_next_message to support holey outboxes by @tillrohrmann in #1640
• Support http1 for request-response protocol deployments by @jackkleeman in #1232
• [restatectl] dump cluster-state by @AhmedSoliman in #1638
• Update rocksdb dependency by @AhmedSoliman in #1653
• CLI should respect that not just virtual objects are stateful by @jackkleeman in #1656
• [reorg] Stage 1 by @AhmedSoliman in #1643
• [reorg] Stage 2 by @AhmedSoliman in #1644
• [Housekeeping] Upgrade toolchain and deps by @AhmedSoliman in #1645
• Fix erroneous enable of test-util by @AhmedSoliman in #1647
• [LogServer] Stub log server service by @AhmedSoliman in #1652
• Replace clean-runner with jlumbroso/free-disk-space by @tillrohrmann in #1657
• Remove empty line which breaks the help message description by @tillrohrmann in #1658
• Only emit admin API incompatibility warning if cluster is running by @tillrohrmann in #1659
• Fix underflow in Millis/Nanos SinceEpoch elapsed by @AhmedSoliman in #1663
• [invoker] Add x-restate-invocation-id header by @igalshilman in #1670
• [Bifrost] minor cleanups by @AhmedSoliman in #1664
• MetadataBuilder to untangle shared metadata construction by @AhmedSoliman in #1665
• Updateable is now Live by @AhmedSoliman in #1668
• Schemas -> Live by @AhmedSoliman in #1671
• Support http2 request-response endpoints (forwardport) by @jackkleeman in #1674
• Remove two default imports from datafusion dependency by @jackkleeman in #1679
• [Bifrost] Eager providers initialization by @AhmedSoliman in #1681
• OpenDB is async and BoxedLiveLoad by @AhmedSoliman in #1682
• [Bifrost] ReplicatedLoglet bifrost stubs by @AhmedSoliman in #1683
• Start ingress only after all partitions have a leader by @tillrohrmann in #1686
• [CLI] Emit correct row count from CLI sql subcommand JSON mode by @pcholakov in #1685
• [Bifrost] Tail semantics change by @AhmedSoliman in #1689
• [Bifrost] Chain and Log metadata improvements by @AhmedSoliman in #1691
• [Bifrost] Logs metadata builder and multi-segment finder by @AhmedSoliman in #1693
• Update leader election protocol by @tillrohrmann in #1694
• Port invoker and a couple of other modules to Hyper 1.0/HTTP 1.0 by @slinkydeveloper in #1716
• Add missing fedora local dev env protobuf dependency by @aradwann in #1728
• Sync nodes configuration if node from the future connects by @tillrohrmann in #1699
• Show invocation completion in CLI by @slinkydeveloper in #1719
• Update aws deps in service-client by @jackkleeman in #1734
• Use http 1.x/hyper 1.x in admin API and part of CLI by @slinkydeveloper in #1733
• Fix bug in IdempotencyId::combine that prevented correct storing of idempotency key in idempotency table by @slinkydeveloper in #1723
• [Bifrost] Decouples loglet dependencies from bifrost by @AhmedSoliman in #1704
• [Bifrost] Decouple loglet errors from bifrost errors by @AhmedSoliman in #1705
• [Bifrost] Base seal tests and implements seal() on memory loglet by @AhmedSoliman in #1717
• [Bifrost] Implements seal() for LocalLoglet by @AhmedSoliman in #1718
• [Bifrost] Init single-node loglets with random loglet ids by @AhmedSoliman in #1720
• [Bifrost] rename bifrost read functions by @AhmedSoliman in #1722
• [Bifrost] read correctly handles seal and tail instability by @AhmedSoliman in #1725
• [Bifrost] Improve merge operator performance by coalescing metadata updates by @AhmedSoliman in #1726
• Update workspace file by @slinkydeveloper in #1738
• Pass through headers in service to service calls by @jackkleeman in #1741
• Bump datafusion dependency to 35.0.0-object-store by @tillrohrmann in #1750
• Replace hyper 0.14 with reqwest and http 1.1 in benchmarks by @tillrohrmann in #1740
• Update metadata-store crate to use prost 0.13, tonic 0.12 and hyper 1.4 by @tillrohrmann in #1746
• Bump prost and tonic dependency to 0.13.1 and 0.12.1 respectively by @tillrohrmann in #1748
• Request metadata updates from peers by @tillrohrmann in #1700
• Update CLI hyper/http dependencies by @jackkleeman in #1739
• [Bifrost] Watch tail updates by @AhmedSoliman in #1743
• [Bifrost] Read stream support multi-segment logs by @AhmedSoliman in #1744
• [Bifrost] Introduces SegmentIndex to identify segments in the log chain by @AhmedSoliman in #1747
• [Bifrost] Basic BifrostAdmin interface by @AhmedSoliman in #1753
• [Bifrost] Remove pin projection from loglet impls by @AhmedSoliman in #1757
• [Bifrost] Loglet read stream with known tail lsn by @AhmedSoliman in #1742
• Push a basic docker image containing the CLI in binary build step by @jackkleeman in #1760
• Fixes for Docker CLI build by @jackkleeman in #1770
• Truncate outbox using an explicit range of sequence ids by @pcholakov in #1762
• [datafusion] Update to the latest version by @igalshilman in #1673
• Fixs SIGSEGV due to missing rocksdb shutdown in leader stepdown test by @AhmedSoliman in #1774
• Run doctests using cargo test by @pcholakov in #1771
• Update otel libraries by @jackkleeman in #1735
• [Bifrost] Bifrost appends will wait for reconfiguration by @AhmedSoliman in #1772
• [Bifrost] DRY loglet tests setup by @AhmedSoliman in #1775
• [Bifrost] handle prefix-trimmed log chains in reads by @AhmedSoliman in #1776
• Make backoff policy in metadata store client configurable by @tillrohrmann in #1761
• Hide the --wipe flag by @muhamadazmy in #1779
• Use configurable retry policy for node networking by @pcholakov in #1773
• Add commands to view and edit/patch service configuration by @slinkydeveloper in #1736
• Configurable network error retry policy by @muhamadazmy in #1782
• Make sure Cargo.lock is updated when committing by @slinkydeveloper in #1787
• Enable SDK-Java/Python/Golang integration tests running by @slinkydeveloper in #1786
• Add PartitionTable by @tillrohrmann in #1702
• Check if all nodes have reported a persisted lsn before trimming the log by @tillrohrmann in #1783
• Make networking handshake timeout configurable by @muhamadazmy in #1796
• [Bifrost] New Appender with Keys support by @AhmedSoliman in #1788
• [Bifrost] BackgroundAppender implementation by @AhmedSoliman in #1789
• [Bifrost] Moving Payload to be an internal bifrost type by @AhmedSoliman in #1792
• [Bifrost] read() now uses LogReadStream internally by @AhmedSoliman in #1794
• [Bifrost] Remove read/read_opt from loglet contract by @AhmedSoliman in #1797
• [Bifrost] Introduce KeyFilter to the read API by @AhmedSoliman in #1798
• Correctly handle gateway errors by @muhamadazmy in #1802
• Implement simple scheduler that watches for leadership and replication by @tillrohrmann in #1780
• [Bifrost] Add WithKeys and BodyWithKeys helper types by @AhmedSoliman in #1808
• [Bifrost] Make StorageEncode object safe by @AhmedSoliman in #1809
• [Bifrost] Mechanical rename to prepare for new payload types by @AhmedSoliman in #1813
• Add SDK-Typescript tests by @slinkydeveloper in #1814
• Perform "just verify" on PRs in CI by @pcholakov in #1824
• Extend restatectl with basic log and node operations by @pcholakov in #1795
• [Bifrost] Major Bifrost write path refactoring by @AhmedSoliman in #1816
• [Bifrost] High-level filtering by keys by @AhmedSoliman in #1817
• [Chore] Remove derive-getters by @AhmedSoliman in #1818
• [Chore] Upgrade derive_more and use it where possible by @AhmedSoliman in #1819
• [Chore] Upgrade dashmap to v6 by @AhmedSoliman in #1820
• [Bifrost] LocalLoglet push-down filtering and new record format by @AhmedSoliman in #1826
• Remove in-memory loglet type from non-test-mode builds by @AhmedSoliman in #1832
• Fix trace HTTP header injection by @jackkleeman in #1836
• Neo invocation status and the Scheduled variant by @slinkydeveloper in #1828
• Flatten effects by @slinkydeveloper in #1837
• Add NeoInvocationStatus integration test run by @slinkydeveloper in #1840
• Allow state to be deleted from non-existent services by @jackkleeman in #1839
• Use net_util::run_hyper_server to run admin http server by @tillrohrmann in #1842
• Fix idempotent send with delay by @slinkydeveloper in #1843
• External metadata store by @muhamadazmy in #1838
• Fix TimerKey by @slinkydeveloper in #1851
• Add Cleaner task to clean completed invocations by @slinkydeveloper in #1845
• Parse the scheduled invocation variant in CLI by @slinkydeveloper in #1848
• Propagate the SpanContext in the Completed status by @slinkydeveloper in #1849
• Update read_pointer when filtering out record by @tillrohrmann in #1860
• Enable SDK-Rust integration tests by @slinkydeveloper in #1865
• Few improvements to state machine module by @slinkydeveloper in #1858
• Expose new state transitions timestamps in SQL by @slinkydeveloper in #1857
• Fix rocksdb stall-detector sleep bug by @AhmedSoliman in #1856
• [Bifrost] Let loglet API support pipelining natively by @AhmedSoliman in #1862
• [TaskCenter] Cleanup of task-center and introduces unmanaged tasks by @AhmedSoliman in #1850
• [Bifrost] Benchpress minor improvements by @AhmedSoliman in #1855
• [Bifrost] Decouple LogletWrapper by @AhmedSoliman in #1861
• Minor tweaks for perf purposes by @AhmedSoliman in #1863
• Control memory-loglet with dedicated feature flag by @AhmedSoliman in #1864
• Fix shutdown warning in partition-store tests by @AhmedSoliman in #1875
• [TaskCenter] PP runtimes with localset support by @AhmedSoliman in #1867
• Update InvocationStatus v2 to use "iS" key prefix by @pcholakov in #1877
• Invoker per PP by @muhamadazmy in #1881
• remove un-necessary clone by @muhamadazmy in #1884
• A few performance improvements by @tillrohrmann in #1874
• Remove the PartitionStorage, StateStorage and StateReader abstraction by @tillrohrmann in #1878
• Resizing PartitionId and LogId and types by @AhmedSoliman in #1879
• [Bifrost] LogletOffset is now u32 by @AhmedSoliman in #1880
• [Bifrost] Replicated loglet params by @AhmedSoliman in #1883
• Replace rocksdb with rust-rocksdb dependency by @tillrohrmann in #1886
• Replace OptimisticTransactionDB with DB and WriteBatchWithIndex by @tillrohrmann in #1887
• Remove CodedErrorResultExt by @slinkydeveloper in #1897
• [Bifrost] Backward-friendly log-server role by @AhmedSoliman in #1895
• MaybeRetryableError as common utility by @AhmedSoliman in #1896
• Side effect retry mechanism by @slinkydeveloper in #1893
• HTTP 1.1 now works with ALPN by @slinkydeveloper in #1900
• [Bifrost] Initial log-server scaffolding by @AhmedSoliman in #1905
• [Bifrost] Replace ErasedInputRecord with Record by @AhmedSoliman in #1906
• [Networking] Introduce top-level message id to correlate messages by @AhmedSoliman in #1912
• [Compat] Fixes for a variety of backward compatibility issues by @AhmedSoliman in #1920
• Upgrade rust version by @aradwann in #1915
• Fix bug wrt cancellation and promise-related entries by @slinkydeveloper in #1921
• Show GetPromise in CLI, together with promise name by @slinkydeveloper in #1919
• Make sure filling datafusion rows won't panic the runtime by @slinkydeveloper in #1923
• [restatectl] Adds simple logs dump for bifrost by @AhmedSoliman in #1917
• Upgrade tokio by @AhmedSoliman in #1926
• Fix bug when filling StartMessage.duration_since_last_stored_entry by @slinkydeveloper in #1931
• Update ClusterMarker doc strings to clarify versioning requirements by @pcholakov in #1925
• Classify Metadata Store failed precondition as successful by @pcholakov in #1932
• Address proc-macro-error advisory RUSTSEC-2024-0370 by @AhmedSoliman in #1934
• [Bifrost] Moves LogletOffset to restate-types by @AhmedSoliman in #1927
• [Networking] Allow sending messages through Outgoing by @AhmedSoliman in #1928
• [Bifrost] Move Record to restate-types by @AhmedSoliman in #1929

New Contributors

• @muhamadazmy made their first contribution in #1779 - welcome to the team!
• @sunng87 made their first contribution in #1634
• @aradwann made their first contribution in #1728

Full Changelog: v1.0.2...v1.1.0