Modernize Bundle Validation CI by Migrating to Testcontainers

[voonhous]

Overview

This discussion proposes a significant modernization of Apache Hudi's bundle validation infrastructure. Currently, our bundle validation process relies on a complex combination of Docker-based shell scripts (ci_run.sh and validate.sh) to verify the integrity and functionality of our release artifacts (Spark, Flink, Utilities, etc.).

This infrastructure is critical as it powers three major workflows:

1. bot.yml: The active CI workflow running on PRs and commits. [MAIN]
2. release_candidate_validation.yml: Validation for release candidates (currently disabled).
3. maven_artifact_validation.yml: Post-release validation for Maven Central artifacts (currently disabled).

The current execution chain involves a GitHub Actions workflow triggering ci_run.sh, which sets up a Docker environment, mounts volumes, and then executes validate.sh inside the container. This script then sequentially runs a series of tests across various bundles.

Current CI Structure

Workflow Execution

• Triggers: The system runs on pushes/PRs to master, release-*, and branch-0.x.
• Matrix Strategy / Coverage: We maintain a comprehensive test matrix covering:
  • Java: 8, 11, 17
  • Scala: 2.12, 2.13
  • Spark: 3.3.x, 3.4.x, 3.5.x, 4.0.0
  • Flink: 1.17 - 1.20, 2.0
• Status: While bot.yml is active for standard CI, the release candidate and Maven artifact validations are currently disabled manually in the YAML files.

Validation Process

Inside the Docker container, validate.sh performs the following validation steps sequentially:

1. Spark & Hadoop MR: Starts Derby/Hive, runs Hive sync, and validates with Spark SQL and HiveQL.
2. Utilities Bundle: Runs HoodieDeltaStreamer and validates output size and content via Spark shell.
3. Utilities Slim Bundle: Validates the slim bundle in conjunction with the Spark bundle.
4. Flink Bundle: Starts a Flink cluster, runs SQL inserts, and validates via a compaction script.
5. Kafka Connect Bundle: Spins up Zookeeper, Kafka, and Schema Registry to test the Hudi Sink connector.
6. Metaserver Bundle: Starts the Metaserver and validates read/write via Spark DataSource.
7. CLI Bundle: Executes a series of Hudi CLI commands to verify table management.

Env Stack

• Orchestration: Bash scripts (ci_run.sh, validate.sh) managing the lifecycle.
• Environment: Custom Docker image (hudi-ci-bundle-validation-base) built in the CI or pulled from Docker Hub with different tags based on different versions.
• Dependencies: Manual management of Derby, Hive, Spark, Flink, Kafka, and Zookeeper lifecycles within the script.

Current Challenges

After analyzing the packaging/bundle-validation/ directory, several pain points are evident:

• Sequential Bottlenecks: Tests run strictly one after another. If the Spark validation takes time, Flink and Kafka tests must wait, extending the feedback loop.
• Fragile Service Management: Services like Hive, Derby, and Zookeeper are started as background processes (&) with PIDs captured for later cleanup (kill $PID). This is prone to zombie processes or port conflicts if a script crashes early.
• Debugging Complexity: "Works on my machine" is hard to achieve. Reproducing a CI failure requires a developer to build the exact Docker image, mount the correct volumes, and run the shell script manually, mimicking the CI environment.
• Opaque Observability: Failures often manifest as a generic "exit code 1" from the shell script. We lack structured JUnit reports for individual validation steps, making it harder to parse which specific test case failed without digging into raw logs.
• Maintenance Overhead: Logic is split between GitHub Actions YAML, ci_run.sh (host setup), and validate.sh (container execution). Adding a new bundle requires updates across multiple files and languages.

Proposed Solution: Migrate to Testcontainers

What is Testcontainers?

Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.

Benefits

• Parallelization: We can run bundle validations concurrently using JUnit 5's parallel execution features. Spark and Flink validations could run simultaneously in isolated containers.
• Developer Experience: Developers can run bundle validation with a standard mvn test command. No need to manually build Docker images or set up volume mounts. <<<<< Writing ITs will not longer be painful and daunting
• Maintainability: Validation logic moves from Bash to Java. We can leverage strong typing, code reuse, and standard Java libraries for assertions and flow control.
• Observability: We gain granular reporting. Each bundle validation becomes a distinct test case in the JUnit report. Failed tests provide standard stack traces and assertion errors.
• Flexibility: We can use WaitStrategies to ensure services like Hive Metastore are fully ready before starting tests, replacing arbitrary sleep commands found in current scripts.
• Consistency: The test environment is defined in code, ensuring that local executions match CI executions perfectly.

Success Metrics / End goal

• Code Quality: Nuke/retire all Bash scripts in favor of typed, testable java code
• Speed up: With parallelization, we should reduce CI runtime.
• Developer Experience: Ability for a contributor to run mvn test -pl packaging/hudi-bundle-validation or write tests just like how they will write unit tests, and be able to run them from their IDE without to switch between terminals and run shell commands.

References

Testcontainers Documentation

JUnit 5 Parallel Execution

Apache Hudi Bundle Validation Readme

Next Steps

If the community is aligned with this proposal, we can start drafting out more concrete plans on how to navigate this migration.

Looking forward to your thoughts!

[cshuo]

+1, the current bundle validation flow is kind of heavy and inconvenient for developers to locating problems when validation fails.

[danny0405]

+1, guess we need to write more codes here to instantiate the engine execution tasks but it should be more easier for debugging.

[xushiyan]

+1. Thanks for raising this @voonhous ! An efficient and easy-to-use test infra will be very helpful, and bundle validation is also a critical piece for releases. There is also a bundle-related RFC #6902

[voonhous]

I've added a issue to track the task progress of this, if there's anyone that wants to contribute or add on to it, feel free to ping me or reply to the issue here: #17961