Adjust pkg/util/compression to a Rust implementation #45519
Draft
+7,192
−202
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Rewrite pkg/util/compression in Rust for improved performance. The Rust implementation becomes the default when CGO is available, with Go implementations as fallback. Changes: - Add Rust compression library with zstd, gzip, zlib, and noop support - Add CGO bindings in pkg/util/compression/impl-rust - Update selector build constraints to use Rust by default - Add invoke tasks for building Rust library (tasks/rust_compression.py) - Fix streaming compressor to track compressed output incrementally - Update serializer tests to work with both implementations The Rust implementation uses: - zstd crate (C bindings) for zstd compression - flate2 crate (pure Rust) for gzip/zlib compression To opt out of Rust compression, build with -tags no_rust_compression. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Results on Apple M1 Max: | Benchmark | Go | Rust | Winner | |-----------|-----|------|--------| | Small (430B) | 1.7 µs | 129 µs | Go 76x faster | | Medium (43KB) | 14.5 µs | 102 µs | Go 7x faster | | Large (4.3MB) | 1,490 µs | 710 µs | Rust 2x faster | | Decompress (43KB) | 6.5 µs | 38.7 µs | Go 6x faster | | Stream (43KB) | 108 µs | 117 µs | ~Same | Memory allocations show Rust is significantly more efficient: - Large payload: Go 4.3MB vs Rust 920B (4700x less) - Medium payload: Go 49KB vs Rust 88B (557x less) The Rust implementation is slower for small/medium payloads due to FFI overhead (Go->Rust->C vs Go->C). Rust only wins on large payloads where compression work dominates over FFI costs. Co-Authored-By: Claude <noreply@anthropic.com>
Performance improvements: - Replace Box<dyn Compressor> with ConcreteCompressor enum (eliminates vtable) - Add #[inline(always)] to hot path functions - Add CompressInto() zero-copy API that writes to caller-provided buffer - Add Go-side buffer pool using sync.Pool - Use zstd::bulk for direct compression Benchmark results (Apple M1 Max): | Payload | Go | Rust Before | Rust After | CompressInto | |-----------|----------|-------------|------------|--------------| | Small | 1,383 ns | 129,000 ns | 1,774 ns | 1,655 ns | | Medium | 10,414ns | 102,000 ns | 7,839 ns | 7,774 ns | | Large | 1,379 µs | 710 µs | 1,493 µs | 1,398 µs | Key improvements: - Small payloads: 76x slower → 1.2x slower (63x improvement) - Medium payloads: 7x slower → 1.3x FASTER - CompressInto uses only 8 bytes allocation regardless of size Co-Authored-By: Claude <noreply@anthropic.com>
… CPU targeting Key changes: - Remove 'thin' feature from zstd crate which was killing performance (thin sets ZSTD_NO_INLINE=1 and DYNAMIC_BMI2=0) - Add .cargo/config.toml with x86-64-v3 baseline for Intel/AMD (AVX2) - Use UnsafeCell for zero-overhead interior mutability in ZstdCompressor - Add stateless and direct zstd FFI paths for benchmarking Performance results (Rust vs Go): - ZSTD: 1.2-2.3x faster across all payload sizes - GZIP: 5-11x faster - ZLIB: 4-11x faster - Zero allocations in Rust (Go gzip/zlib allocate ~814KB/call) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…sion - Add sync.Mutex to RustCompressor for thread-safe concurrent access - Add runtime.SetFinalizer to prevent memory leaks if Close() not called - Cache ContentEncoding() to avoid CGO overhead on repeated calls - Fix buffer ownership bug in impl-zstd-nocgo: verify compressed data is in original buffer's backing array, not a reallocated slice - Unify thread-safety documentation across Rust, C header, and Go: clarify that compressor handles require external synchronization (Go wrapper provides this via mutex) - Add concurrent compression tests and finalizer tests - Add release notes for the Rust compression feature Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix buffer ownership bug in impl-zstd (same fix applied to zstd-nocgo): check capacity before compression and verify backing array after - Make CompressBound lock-free by computing bounds in Go using formulas that exactly match the Rust implementations (avoids CGO + mutex overhead) - Add reusable headerBuf in payloadsBuilderV3 to avoid allocation per protobuf field header during payload serialization - Add unit tests for impl-zstd covering compression, buffer ownership, error cases, and stream compression Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
- Add test-only decompression utilities in pkg/util/compression/testutil - Update serializer tests to use CompressInto + testutil.Decompress - Update stream tests (compressor_test, column_test) for new API - Update metrics tests (events, service_checks, sketch_series) - Add fuzz tests for round-trip verification across Go/Rust implementations - Adjust test parameters for Rust compression efficiency differences Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
github-actions
bot
added
team/agent-runtimes
long review
The Rust compression library is now automatically built as part of `dda inv agent.build`. This ensures the library is available when CGO is enabled, without requiring manual build steps or CI changes. Added --exclude-rust-compression flag to skip the Rust build when needed (falls back to Go implementations via build tags). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Contributor
Go Package Import DifferencesBaseline: e1e8408
|
Add Rust compression library build step to: - dogstatsd.build - trace-agent.build - process-agent.build - cluster-agent.build - security-agent.build - system-probe.build Each build task now automatically builds the Rust compression library before compiling, ensuring the library is available when CGO is enabled. All tasks support --exclude-rust-compression flag to skip Rust build. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add no_rust_compression to ALL_TAGS as a valid build tag for fallback - Add Rust compression build step to check_mod_tidy in tasks/go.py - Add Rust compression build step to KMT CI jobs (common.yml, security_agent.yml, system_probe.yml) - Add Rust compression build step to tasks/ebpf.py for verifier calculator - Add Rust compression build step to tasks/kernel_matrix_testing/stacks.py This ensures Rust compression is used everywhere by building the library before any Go code that depends on it. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Contributor
Gitlab CI Configuration Changes
|
| Removed | Modified | Added | Renamed |
|---|---|---|---|
| 0 | 91 | 1 | 0 |
ℹ️ Diff available in the job log.
Integrate the Rust compression library build into all remaining invoke tasks that compile Go code which may use the compression package: - omnibus.py: Main packaging build and build_repackaged_agent - otel_agent.py: OTel agent build - sbomgen.py: SBOM generator build - loader.py: Loader build - full_host_profiler.py: Full host profiler build - installer.py: Installer build - privateactionrunner.py: Private action runner build - cws_instrumentation.py: CWS instrumentation build - cluster_agent_cloudfoundry.py: Cluster agent for Cloud Foundry build Each task now calls rust_compression_build(ctx, release=True) before the Go build, with an exclude_rust_compression parameter to optionally skip when not needed. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add cargo command mocks to omnibus unit tests so they properly simulate the Rust compression build step - Add Rust compression build to tests_serverless-init and tests_serverless-init-logs CI jobs before running go test Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The build_sysprobe_binary task is called by omnibus for Linux builds but was missing the Rust compression build step, causing linker errors. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove cdylib from Cargo.toml to only build static library, avoiding runtime shared library loading issues - Add Rust compression build to oracle.test task Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit ensures Rust compression is built before any Go compilation that may link against the libdatadog_compression static library. Added to: - tasks/gotest.py: Core test runner used for unit tests - tasks/gointegrationtest.py: Integration test runner - tasks/new_e2e_tests.py: E2E test binaries and test runs - tasks/bench.py: Benchmark builds - tasks/fakeintake.py: Fakeintake build and tests All tasks follow the same pattern: building Rust compression at the start unless exclude_rust_compression=True is passed. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The integration test CI jobs were failing because Rust was not being built before running tests. Added rust-compression.build to: - .gitlab/test/integration_test/linux.yml: Both docker and trace agent integration tests - tasks/winbuildscripts/pre-go-build.ps1: Windows pre-build script for integration tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The omnibus build was failing because the CGO_LDFLAGS environment variable set by omnibus did not include the path to the Rust compression library. This caused the Go linker to not find libdatadog_compression.a during the build. Updated the following omnibus software configs to include the Rust library path in CGO_LDFLAGS: - datadog-agent.rb - datadog-dogstatsd.rb - datadog-iot-agent.rb - datadog-otel-agent.rb Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The system-probe and security-agent KMT functional test preparation jobs need to build the Rust compression library before running kmt.prepare, which compiles test binaries that link against libdatadog_compression. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The otel-agent.integration-test job links against libdatadog_compression and needs the Rust library built before running the tests. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit adds rust-compression.build --release to all CI jobs that compile Go code which may link against libdatadog_compression. This includes: Binary builds: - dogstatsd (static and regular, x64 and arm64) - iot agent (x64 and arm64) - otel agent - cluster agent - cluster agent cloudfoundry - cws instrumentation - host profiler - system probe - fakeintake Package builds: - agent packages (omnibus) - iot packages - dogstatsd packages - otel packages - heroku packages - installer packages Tests: - Linux unit tests - macOS unit tests - Oracle functional tests - E2E test binaries Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The docker-in-docker runners (micro-vms) may not have Rust installed. Add a reusable anchor to install Rust via rustup if cargo is not found. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…path mismatch The Rust compression library is built in /go/src/... but omnibus copies the source to /omnibus/src/... causing CMake cache conflicts. Cleaning the target directory before building ensures a fresh build without cached paths. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… conflict The omnibus build internally calls cargo to build the rust library after copying the source to /omnibus/src/. Running rust-compression.build in the CI script creates CMake cache files with paths pointing to /go/src/, which causes conflicts when omnibus tries to rebuild in the copied location. The fix removes the redundant rust-compression.build call from omnibus jobs while keeping the rm -rf to ensure no stale cache gets copied. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…runners The libz-ng-sys crate requires cmake to build. Docker-in-docker runners don't have cmake pre-installed, so add it to the dependency installation step. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
blt
added a commit
that referenced
this pull request
Jan 27, 2026
This commit is motivated by the potential end state in spike PR #45519. That is, if we want to make non-trivial changes to this package systematic testing of the package is warranted. I have introduced tests to confirm that for all the strategies supported by this package: - the identity `b == decompress(compress(b))` holds * that this is true also for zstd cgo/nocgo crossed - empirical confirmation of the `CompressBound` calculations - tests for resource management by the streaming compression This commit includes fixes for the following: - zlib CompressBound: Changed from 13 to 16 bytes overhead - gzip CompressBound: Changed from 18 to 33 bytes overhead - zstd CGO: Fixed write-after-close bug by adding a wrapper with state tracking This commit likewise includes benchmarks for the package.
The rust-compression library may be built in /go/src/... before omnibus copies the source to /omnibus/src/..., causing CMake cache path conflicts. Adding delete "pkg/util/compression/rust/target" in each software definition ensures a clean build without stale CMake cache files. Affected software: - datadog-agent - datadog-dogstatsd - datadog-iot-agent - datadog-otel-agent - installer Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The rust-compression library requires Rust toolchain and cmake to build. Adding these dependencies to the Dockerfile. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
blt
added a commit
that referenced
this pull request
Jan 27, 2026
This commit is motivated by the potential end state in spike PR #45519. That is, if we want to make non-trivial changes to this package systematic testing of the package is warranted. I have introduced tests to confirm that for all the strategies supported by this package: - the identity `b == decompress(compress(b))` holds * that this is true also for zstd cgo/nocgo crossed - empirical confirmation of the `CompressBound` calculations - tests for resource management by the streaming compression This commit includes fixes for the following: - zlib CompressBound: Changed from 13 to 16 bytes overhead - gzip CompressBound: Changed from 18 to 33 bytes overhead - zstd CGO: Fixed write-after-close bug by adding a wrapper with state tracking Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This is an experiment. This PR will only ever be marked as draft. There is no path for this specific PR to ever merge to main.
The goal of this experiment is to understand how / if the Agent project is amenable to mechanical translation between languages.
pkg/util/compressionwas chosen because it has a minimal interface, well-defined semantics for that interface and is relatively central to how data moves through the system. Sub-goal: if Agent project is amenable to mechanical translation can we improve performance by this transformation?Micro-benchmark results:
Changes Introduced
As noted above, this PR has too many incorporated changes to review. If the results here are compelling the sub-PRs to make are:
b = decompress(compress(b))property fuzzing.And more besides I'm sure, these are just the high-level points to hit.
Additional Notes
There is a workflow here that is semi-mechanical in a fashion that @preinlein and I have been experimenting with in DataDog/lading, see these skills.
Also the Rust build is just sorta jammed into this PR. There will probably be some CI fiddling tacked onto the commits I am opening this draft with.
Open Questions