Skip to content

Add support for build templates (and pixi) #839

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
pditommaso merged 56 commits into from
Dec 10, 2025
Merged

Add support for build templates (and pixi) #839

pditommaso merged 56 commits into from
Dec 10, 2025

Conversation

@pditommaso
Copy link
Collaborator

@pditommaso pditommaso commented May 17, 2025 •
edited
Loading

Summary

This PR introduces support for multi-stage build templates (Pixi and Micromamba v2), updates libseqera dependencies, and includes various documentation improvements and bug fixes.

Key Changes

Build Template Support

Added buildTemplate parameter to support new container build strategies:

Template Description
conda/micromamba:v1 Default Micromamba 1.x template (legacy)
conda/micromamba:v2 Micromamba 2.x with multi-stage builds
conda/pixi:v1 Pixi package manager with multi-stage builds for optimized image sizes
cran/installr:v1 R/CRAN package builds using installr

Multi-stage templates (conda/micromamba:v2 and conda/pixi:v1) create optimized container images by separating the build environment from the final runtime environment, resulting in smaller container images that only contain installed packages and runtime dependencies.

Documentation

  • Added comprehensive API examples for Pixi and Micromamba v2 templates in docs/api.md
  • Added ADR documenting design decisions for multi-stage build templates

API Changes

New fields added to the container provisioning API (/v1alpha2/container):

Field Description
buildTemplate The build template to use. Values: conda/pixi:v1, conda/micromamba:v2, cran/installr:v1
pixiOpts Pixi build options (when using conda/pixi:v1 template)
pixiOpts.pixiImage Docker image for Pixi package manager (default: ghcr.io/prefix-dev/pixi:0.59.0-noble)
pixiOpts.baseImage Base image for final stage of multi-stage builds (default: ubuntu:24.04)
pixiOpts.basePackages Additional packages to install in final image
pixiOpts.commands Custom commands to run during build

Implementation Details

  • New utility methods in ContainerHelper:
    • containerFileFromRequest() - Routes to appropriate template based on buildTemplate parameter
    • Template validation and error handling for unsupported package types
  • New Dockerfile/Singularity generation methods in DockerHelper:
    • condaFileToDockerFileUsingPixi()
    • condaFileToSingularityFileUsingPixi()
    • condaFileToDockerFileUsingMicromamba()
    • condaFileToSingularityFileUsingMicromamba()

Test Plan

  • Verify Pixi template builds create optimized multi-stage containers
  • Verify Micromamba v2 template builds work correctly
  • Test backward compatibility with existing container builds (no template specified)
  • Validate updated libseqera dependencies work as expected
  • Run full test suite: ./gradlew test
  • Build and test container image: ./gradlew jibDockerBuild
  • Test Singularity builds with new templates

pditommaso added 2 commits May 16, 2025 21:07
@pditommaso
[release] bump version 1.19.7
9637ab1 
Signed-off-by: Paolo Di Tommaso <[email protected]>
@pditommaso
Initial pixi support
413d730 
Signed-off-by: Paolo Di Tommaso <[email protected]>
@pditommaso pditommaso marked this pull request as draft May 17, 2025 01:08
@pditommaso
Add singularity file
7c3da4e 
Signed-off-by: Paolo Di Tommaso <[email protected]>
This was referenced May 18, 2025
Open
Closed
ewels
ewels reviewed May 27, 2025
View reviewed changes
@munishchouhan munishchouhan self-assigned this Nov 11, 2025
@robnewman

This comment has been minimized.

Sign in to view
@munishchouhan

This comment has been minimized.

Sign in to view
@munishchouhan

This comment has been minimized.

Sign in to view
@pditommaso
[release] bump version 1.31.0-B3
7f107db 
Signed-off-by: Paolo Di Tommaso <[email protected]>
@munishchouhan

This comment has been minimized.

Sign in to view
@munishchouhan

This comment has been minimized.

Sign in to view
@munishchouhan

This comment has been minimized.

Sign in to view
@pditommaso

This comment has been minimized.

Sign in to view
@munishchouhan

This comment has been minimized.

Sign in to view
@munishchouhan

This comment has been minimized.

Sign in to view
@pditommaso

This comment has been minimized.

Sign in to view
@munishchouhan

This comment has been minimized.

Sign in to view
@munishchouhan
fixed pixi singularity build
0386b4f 
Signed-off-by: munishchouhan <[email protected]>
@munishchouhan

This comment has been minimized.

Sign in to view
@munishchouhan

This comment has been minimized.

Sign in to view
@pditommaso

This comment has been minimized.

Sign in to view
@munishchouhan

This comment has been minimized.

Sign in to view
@pditommaso

This comment has been minimized.

Sign in to view
munishchouhan and others added 9 commits December 5, 2025 20:24
@munishchouhan
fixed micromamba-v2 singularity build
dc3b650 
Signed-off-by: munishchouhan <[email protected]>
@munishchouhan
added docs
ab6a6ed 
Signed-off-by: munishchouhan <[email protected]>
@munishchouhan
[release] test 1.31.0-B4
f575c78 
Signed-off-by: munishchouhan <[email protected]>
@munishchouhan
fixed test
ce07fa5 
Signed-off-by: munishchouhan <[email protected]>
@pditommaso @claude
Document tar requirement for Singularity multi-stage builds
3fcafd7 
Clarify that the tar utility requirement for conda/pixi/v1 and
conda/micromamba/v2 templates only applies to Singularity builds.
This is due to proot limitations in Singularity multi-stage builds
which cannot copy directory structures directly between stages.

Updated documentation in:
- ADR: Added explanation of proot limitation
- API docs: Clarified Singularity-only requirement
- CLI docs: Added Singularity build requirements section

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@pditommaso
[release]
f2c3967 
Signed-off-by: Paolo Di Tommaso <[email protected]>
@munishchouhan
Merge branch 'master' into build-template
1fd8ced
@pditommaso
Revert version
c1ec20a 
Signed-off-by: Paolo Di Tommaso <[email protected]>
@pditommaso @claude
Change build template syntax to <scope>/<name>:<version> format
a03e1cb 
Update template identifiers to use colon before version:
- conda/pixi/v1 -> conda/pixi:v1
- conda/micromamba/v1 -> conda/micromamba:v1
- conda/micromamba/v2 -> conda/micromamba:v2
- cran/installr/v1 -> cran/installr:v1

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@pditommaso
Copy link
Collaborator Author

pditommaso commented Dec 10, 2025

Ready ✅

@pditommaso

This comment was marked as outdated.

Sign in to view
@claude

This comment was marked as off-topic.

Sign in to view
@pditommaso

This comment was marked as off-topic.

Sign in to view
@pditommaso pditommaso merged commit 21e9e42 into master Dec 10, 2025
5 checks passed
@pditommaso pditommaso deleted the build-template branch December 10, 2025 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ewels ewels ewels left review comments

@cristianrcv cristianrcv Awaiting requested review from cristianrcv

@stefanoboriero stefanoboriero Awaiting requested review from stefanoboriero

Assignees

@munishchouhan munishchouhan

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@pditommaso @robnewman @munishchouhan @ewels