Optimize CI with reusable image workflow #4782

Deepam02 · 2025-12-04T19:31:26Z

What this PR does / why we need it:

This PR optimizes the CI pipeline by extracting Docker image building into a centralized reusable workflow. Previously, each E2E test job built images independently, resulting in redundant builds and wasted resources.

Changes:

Created images.yaml as a reusable workflow that builds all Volcano Docker images once
Saves built images as artifacts
Updated all 10 E2E workflows to download and load pre-built images instead of rebuilding
Uses standard docker save/docker load commands for artifact handling

Benefits:

Eliminates redundant image builds across parallel jobs
Significantly reduces CI execution time and resource usage
Faster feedback loop when retriggering failed pipelines
Images are built once and reused across all test jobs

Which issue(s) this PR fixes:

Fixes #4766

Special notes for your reviewer:

The implementation follows GitHub's reusable workflow pattern. All E2E workflows now have a needs: build-images dependency and download artifacts before running tests. The existing test infrastructure remains unchanged - only the image provisioning mechanism has been optimized.

gemini-code-assist · 2025-12-04T19:31:32Z

Note

Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported.

volcano-sh-bot · 2025-12-04T19:31:38Z

Welcome @Deepam02! It looks like this is your first PR to volcano-sh/volcano 🎉

Copilot

Pull request overview

This PR optimizes the CI pipeline by introducing a centralized, reusable workflow for Docker image building, eliminating redundant image builds across parallel E2E test jobs. The optimization reduces CI execution time and resource consumption by building all Volcano images once and distributing them as artifacts.

Key Changes:

Created images.yaml reusable workflow that builds all four Volcano Docker images once and uploads them as artifacts with 5-day retention
Updated 10 E2E workflow files to depend on the centralized build job and load pre-built images instead of rebuilding
Modified test execution commands to use the common hack/run-e2e-kind.sh script with E2E_TYPE parameters

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`.github/workflows/images.yaml`	New reusable workflow that builds and saves all Volcano Docker images as artifacts
`.github/workflows/e2e_vcctl.yaml`	Added build-images dependency, downloads artifacts, and updated test execution to use run-e2e-kind.sh
`.github/workflows/e2e_spark.yaml`	Added build-images dependency, downloads artifacts, modified image loading for minikube environment
`.github/workflows/e2e_sequence.yaml`	Added build-images dependency, downloads artifacts, and updated test execution to use run-e2e-kind.sh
`.github/workflows/e2e_scheduling_basic.yaml`	Added build-images dependency, downloads artifacts, and updated test execution to use run-e2e-kind.sh
`.github/workflows/e2e_scheduling_actions.yaml`	Added build-images dependency, downloads artifacts, and updated test execution to use run-e2e-kind.sh
`.github/workflows/e2e_parallel_jobs.yaml`	Added build-images dependency, downloads artifacts, and updated test execution to use run-e2e-kind.sh
`.github/workflows/e2e_hypernode.yaml`	Added build-images dependency, downloads artifacts, and updated test execution to use run-e2e-kind.sh
`.github/workflows/e2e_dra.yml`	Added build-images dependency, downloads artifacts, and updated test execution to use run-e2e-kind.sh with DRA feature gate
`.github/workflows/e2e_cronjob.yaml`	Added build-images dependency, downloads artifacts, and updated test execution to use run-e2e-kind.sh
`.github/workflows/e2e_admission.yaml`	Added build-images dependency and downloads artifacts for both admission policy and webhook test jobs

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-04T19:36:47Z

.github/workflows/e2e_spark.yaml

+        # Use git SHA as TAG to match pre-built images
+        export TAG=$(git rev-parse --verify HEAD)
+        
+        make TAG=${TAG} update-development-yaml


[nitpick] The TAG is being set here using git rev-parse --verify HEAD, but the workflow still has an earlier step (lines 39-48) that sets TAG from .release-version or defaults to 'latest'. While that earlier step's output is now unused (overridden here), consider removing that obsolete step in a future cleanup to avoid confusion about where TAG comes from.

JesseStutler · 2025-12-05T06:11:26Z

/cc @hajnalmt

hajnalmt

Thank you for the contribution @Deepam02 !

Please find my comments below and let's see this in action if you managed to address them😀
Also please squash your commits, there is a merge commit at end of your commit history. Rebase your local branch on the master and do not merge it directly.

Thank you once more!

.github/workflows/images.yaml

hajnalmt · 2025-12-05T09:01:35Z

.github/workflows/images.yaml

+        with:
+          name: volcano-images
+          path: images
+          retention-days: 5


I was wondering if this is maybe not enough, but let's stay with 5, you are probably right.

.github/workflows/e2e_admission.yaml

.github/workflows/e2e_spark.yaml

.github/workflows/e2e_dra.yml

Signed-off-by: Deepam Goyal <[email protected]>

Deepam02 · 2025-12-08T05:56:02Z

Hi @hajnalmt
I have made all the suggested changes, squashed the commits, and rebased.
Sorry for the delay in getting this done, please review it again when you get time.

hajnalmt

/ok-to-test

Thank you for the change!
I had some minor comments still. Please resolve them too, but we can start the testing I think.

Makefile

hajnalmt

/hold

@Deepam02
As the CI started to run I realized that this current structure is not enough as now every workflow triggers it's own image building.

What we want to achieve is that every workflow shall use the output artifact of the images workflow. This means that we need a separate e2e_test workflow that calls the current e2e workflows one by one (paralelly) after the images workflow.

This also means the modification of the current e2e workflows to be a reusable ones.
so we need to modify the on part to be:

on:
  workflow_call:

instead of:

on:
  push:
    branches:
      - master
    tags:
  pull_request:

And this new e2e_test workflow shall have this trigger instead.

I hope it's not a big a change. I completely missed this in the original specification.

Signed-off-by: Deepam Goyal <[email protected]>

Deepam02 · 2025-12-08T13:15:37Z

@hajnalmt

I've restructured the workflows as suggested:

Created a new e2e.yaml orchestrator that calls images.yaml first, then triggers all E2E workflows in parallel

CI should be green now. Please review when you have a chance.

hajnalmt · 2025-12-08T13:25:21Z

/unhold

Thank you! Let's see :)

hajnalmt

/test all

volcano-sh-bot · 2025-12-08T13:27:07Z

@hajnalmt: No presubmit jobs available for volcano-sh/volcano@master

Details

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hajnalmt · 2025-12-08T13:35:48Z

/retest

hajnalmt · 2025-12-08T13:40:42Z

/ok-to-test cancel

hajnalmt · 2025-12-08T13:44:25Z

/cc @JesseStutler

Can you trigger testing somehow on this one? We probably need to remove the ok-to-test label and readd it since the workflow structure changed.

hajnalmt · 2025-12-08T13:53:22Z

/rerun-all

JesseStutler · 2025-12-09T03:26:01Z

/cc @JesseStutler

Can you trigger testing somehow on this one? We probably need to remove the ok-to-test label and readd it since the workflow structure changed.

/ok-to-test label is enough

JesseStutler · 2025-12-09T03:29:15Z

.github/workflows/e2e_cronjob.yaml

        with:
          go-version: 1.24.x

-      - name: Install musl


So currently we don't need musl now? @Deepam02 @hajnalmt

We are currently not using musl at all.

The Dockefiles are not processing the CC variable as arguments so it will use gcc.
scheduler Dockerfile

It's not even passed to the images target:
images target
It was just blindly copied each time a new CI job was introduced I guess.

Probably when Monokaix introduced the custom plugins:
https://github.com/volcano-sh/volcano/tree/master/example/custom-plugin
It was added everywhere for some reason, but It's hard to understand the reason behind this now.

Yes, so should we keep the musl if we don't know the reason?

Since we are not using it, we should delete the step I think. It just occupies the space and CI time for no reason.

JesseStutler · 2025-12-09T03:32:45Z

.github/workflows/e2e_hypernode.yaml

-      - master
-    tags:
-  pull_request:
+  workflow_call:


Sorry I'm not so familiar with workflow_call, so all the CIs can stil be triggered throught the e2e.yaml, right?

Yes hopefully, if we do a retrigger with retest it will only retrigger the failed tests too.
But some testing probably needs to be done on this to be sure.

But I found that we still have these old pending checks waiting to be reported, but we already have execute these CIs, we should not execute them twice, right?

This is there because the master branch triggers them. It should disappear I think after we manage to merge this PR.

JesseStutler · 2025-12-09T08:43:41Z

Seems that all the CIs have triggered but some CIs failed such as e2e-spark and e2e-vcctl, some failed CIs event can't load images @Deepam02 But why we still have pending checks here?

Shouldn't we already have some CIs passed and don't need to execute the independent old CIs again?

JesseStutler · 2025-12-09T08:51:22Z

What's more, you still have built all the images in each workflow:

But what we want is that we just need to build the images only once, and load to the kind clusters in each workflow. @Deepam02

hajnalmt

Currently the images are rebuilt because we checkout the code in the e2e_* again. Basically we are checking everything out twice, so all the timestamps differ from the ones in the build_images step. You shall move everything to the e2e.yaml which can be done at the invoking job.

hajnalmt · 2025-12-09T09:07:22Z

.github/workflows/e2e_hypernode.yaml

-      - master
-    tags:
-  pull_request:
+  workflow_call:


This is there because the master branch triggers them. It should disappear I think after we manage to merge this PR.

hajnalmt · 2025-12-09T09:08:08Z

.github/workflows/e2e_dra.yml

-          tar -xf musl-1.2.1.tar.gz && cd musl-1.2.1
-          ./configure
-          make && sudo make install
      - name: Checkout code


Checkout code and Install dependencies can be moved to e2e.yaml too! We don't need them in every e2e test now.

volcano-sh-bot · 2025-12-09T09:17:34Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign wangyang0616 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hajnalmt · 2025-12-09T09:51:29Z

Makefile


 images:
 	for name in controller-manager scheduler webhook-manager agent; do\
 		docker buildx build -t "${IMAGE_PREFIX}/vc-$$name:$(TAG)" . -f ./installer/dockerfile/$$name/Dockerfile --output=type=${BUILDX_OUTPUT_TYPE} --platform ${DOCKER_PLATFORMS} --build-arg APK_MIRROR=${APK_MIRROR} --build-arg OPEN_EULER_IMAGE_TAG=${OPEN_EULER_IMAGE_TAG}; \


Alternatively to not rebuild the images, you can add a check here, to not rebuild if the tag already exists.
You can introduce a new variable to force the image build (like FORCE_REBUILD) which is true by default and start the target with false make images FORCE_REBUILD=false in the e2e tests, so you are backwards compatible with the old make target.

lekaf974 · 2025-12-17T18:05:07Z

.github/workflows/images.yaml

Should this file name be more explicit docker_images.yaml for example ?

Copilot AI review requested due to automatic review settings December 4, 2025 19:31

volcano-sh-bot requested review from Yikun and kingeasternsun December 4, 2025 19:31

volcano-sh-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 4, 2025

Copilot started reviewing on behalf of Deepam02 December 4, 2025 19:31 View session

volcano-sh-bot added the do-not-merge/contains-merge-commits label Dec 4, 2025

Copilot finished reviewing on behalf of Deepam02 December 4, 2025 19:36

Copilot AI reviewed Dec 4, 2025

View reviewed changes

Deepam02 mentioned this pull request Dec 4, 2025

Improve ci with reusable workflows #4766

Open

JesseStutler self-requested a review December 5, 2025 06:10

volcano-sh-bot requested a review from hajnalmt December 5, 2025 06:11

hajnalmt suggested changes Dec 5, 2025

View reviewed changes

hajnalmt reviewed Dec 5, 2025

View reviewed changes

.github/workflows/e2e_spark.yaml Outdated Show resolved Hide resolved

hajnalmt reviewed Dec 5, 2025

View reviewed changes

.github/workflows/e2e_dra.yml Outdated Show resolved Hide resolved

Optimize CI with reusable image workflow

4ceb1a3

Signed-off-by: Deepam Goyal <[email protected]>

Deepam02 force-pushed the optimize-ci-image-building branch from 87843e1 to 4ceb1a3 Compare December 8, 2025 05:49

volcano-sh-bot removed the do-not-merge/contains-merge-commits label Dec 8, 2025

hajnalmt reviewed Dec 8, 2025

View reviewed changes

Makefile Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

volcano-sh-bot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Dec 8, 2025

hajnalmt suggested changes Dec 8, 2025

View reviewed changes

volcano-sh-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 8, 2025

Deepam02 added 2 commits December 8, 2025 18:37

Restructure workflows to centralize image building

cd5b34e

Signed-off-by: Deepam Goyal <[email protected]>

Remove unnecessary --format oci flag from docker save

4d9dad3

Signed-off-by: Deepam Goyal <[email protected]>

volcano-sh-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 8, 2025

hajnalmt approved these changes Dec 8, 2025

View reviewed changes

JesseStutler reviewed Dec 9, 2025

View reviewed changes

hajnalmt suggested changes Dec 9, 2025

View reviewed changes

hajnalmt reviewed Dec 9, 2025

View reviewed changes

JesseStutler added the area/ci Issues or PRs related to volcano CIs label Dec 10, 2025

lekaf974 reviewed Dec 17, 2025

View reviewed changes

.github/workflows/images.yaml

Copy link

Contributor

lekaf974 Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this file name be more explicit docker_images.yaml for example ?

Optimize CI with reusable image workflow #4782

Are you sure you want to change the base?

Optimize CI with reusable image workflow #4782

Uh oh!

Conversation

Deepam02 commented Dec 4, 2025

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Uh oh!

gemini-code-assist bot commented Dec 4, 2025

Uh oh!

volcano-sh-bot commented Dec 4, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

JesseStutler commented Dec 5, 2025

Uh oh!

hajnalmt left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Deepam02 commented Dec 8, 2025

Uh oh!

hajnalmt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hajnalmt left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Deepam02 commented Dec 8, 2025

Uh oh!

hajnalmt commented Dec 8, 2025

Uh oh!

hajnalmt left a comment

Choose a reason for hiding this comment

Uh oh!

volcano-sh-bot commented Dec 8, 2025

Uh oh!

hajnalmt commented Dec 8, 2025

Uh oh!

hajnalmt commented Dec 8, 2025

Uh oh!

hajnalmt commented Dec 8, 2025

Uh oh!

hajnalmt commented Dec 8, 2025

Uh oh!

JesseStutler commented Dec 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hajnalmt left a comment •

edited

Loading

hajnalmt left a comment •

edited

Loading

hajnalmt Dec 9, 2025 •

edited

Loading

hajnalmt Dec 9, 2025 •

edited

Loading

hajnalmt Dec 9, 2025 •

edited

Loading