feat(artifacts): Object storage integration and artifacts key

[KellyMerrick]

Previous branch with discussions: #1294

Overview

1. Introduce a pluggable object storage subsystem that Vela can use for build artifacts (test results, screenshots, videos, etc.).
2. Add a first‑class artifact model at the database and API layers, plus compiler/YAML support for an artifacts key on steps.
3. Expose HTTP endpoints for creating and listing artifacts, plus admin endpoints for managing storage (buckets, presigned URLs).
4. Wire storage + artifacts into the router, middleware, CLI flags, and local dev environment so the feature can be exercised end‑to‑end.
5. Proposal feat(proposal): native test report integration community#1022

---

High‑Level Architecture

1. Storage subsystem

Packages & files (high level)

• storage/
  • context.go, setup.go, service.go, storage.go (+ tests)
  • flags.go (+ tests) for CLI/env configuration
• storage/minio/
  • Bucket operations: create_bucket.go, bucket_exists.go, list_bucket.go, get_bucket.go
  • Object operations: upload.go, list_objects.go, stat_object.go, presigned_get_object.go
  • Driver config / enablement: opts.go, storage_enable.go, minio.go
  • Test data: test_data/create_bucket.json, test_data/test.xml
• constants/driver.go, constants/filetypes.go, constants/table.go

Design

• A new storage service interface is introduced (see storage/service.go / storage/storage.go), following the same Service + Setup + Context pattern as other Vela subsystems.
• The initial implementation is a MinIO/S3‑compatible driver, isolated under storage/minio/.
• Storage enablement is driven by configuration + feature flag:
  • The driver and settings are wired via storage/flags.go and storage/setup.go.
  • Middleware injects a storage-enable flag and a storage client into the Gin context.
• Common constants for:
  • Drivers (e.g. MinIO),
  • File types for artifacts,
  • Table names for the new artifact table.

2. Artifact data model

Database layer

• New database/artifact/ package:
  • CRUD, list, count, and index operations:
    • create.go, get.go, get_build.go, list.go, list_build.go, count.go, update.go, delete.go, index.go, opts.go, table.go
  • Full test suite mirroring the above (*_test.go).
• New persistence model under database/types/:
  • artifact.go
• database/database.go, database/interface.go, and database/resource.go updated to:
  • Register the new artifact service with the main database interface.
  • Expose it via the existing resource wiring.
• database/testutils/api_resources.go updated to provide helpers for the new resource.

Conceptually

• Artifacts represent build‑scoped files stored in object storage (e.g. JUnit XML, Cypress screenshots, videos).
• Each artifact record tracks metadata (build ID, file name, file type, size, object path, timestamps) and is the link between the build in the DB and the object in storage.

3. API + routing surface

API packages

• Storage admin + info
  • api/admin/storage.go: admin handlers for bucket creation and presigned URLs for objects.
  • api/storage/doc.go + api/storage/storage.go: public storage handlers (e.g., storage info / health).
• Artifacts
  • api/artifact/create.go
  • api/artifact/get.go
  • api/artifact/list.go
• API types
  • api/types/artifact.go
  • api/types/storage.go
  • api/types/storage_info.go
  • Minor updates in api/types/pipeline.go / tests to incorporate the artifact‑related shape.

Router & middleware

• New or updated router files:
  • router/storage.go — routes for storage info and/or admin integration.
  • router/artifact.go — routes for artifact create/list/get.
  • router/admin.go — wires the new admin storage operations.
  • router/build.go — extended to add build‑scoped artifact routes.
• Middleware:
  • router/middleware/storage.go (+ tests): constructs the storage service, sets storage-enable, and injects it into Gin.
  • router/middleware/artifact/context.go, artifact/artifact.go: helpers for binding artifact resources to requests.

Swagger / HTTP endpoints

• Storage admin:
  • PUT /api/v1/admin/storage/bucket — create a bucket.
  • GET /api/v1/admin/storage/presign — generate a presigned URL for a given bucket + object.
• Artifacts:
  • New repo/build‑scoped create, get, and list endpoints under artifacts (see router + API handlers).
  • These endpoints are backed by the new artifact database package and the storage service (for presigned URLs).

4. Compiler & pipeline types

• compiler/types/pipeline/artifact.go (+ tests) and compiler/types/yaml/artifacts.go:
  • Introduce an artifacts concept into the compiler/pipeline types.
• compiler/types/pipeline/container.go, compiler/types/yaml/step.go, compiler/types/yaml/step_test.go:
  • Allow steps to declare artifacts configuration.
• compiler/native/validate.go (+ tests):
  • Validation extended/adjusted to understand the new artifacts key usage in the pipeline YAML.

This sets up the server side so that pipelines can emit artifact metadata that is persisted and linked to stored objects in the configured bucket.

5. Configuration & local dev

• cmd/vela-server/main.go, metadata.go, server.go:
  • Wire storage flags and setup into server startup.
• storage/flags.go, storage/setup.go:
  • Define CLI/env flags for storage configuration (driver, connection details, etc.).
  • Build the storage.Service and register it with the router via middleware.
• docker-compose.yml and nginx.conf:
  • Add MinIO to the local dev stack.
  • Configure NGINX / hostnames so minio is reachable from the server and UI.
• DOCS.md:
  • Document the MinIO dev setup (e.g. adding minio to /etc/hosts) and reference storage + artifacts in the setup flow.

---

Testing

• New unit tests added for:
  • Storage driver operations (storage/minio/*_test.go).
  • Storage wiring (storage/context_test.go, storage/setup_test.go, storage/storage_test.go, storage/flags_test.go).
  • Artifact CRUD, list, count, and indexing (database/artifact/*_test.go).
  • New database resource wiring (database/resource_test.go).
  • Router + middleware behavior for storage and artifacts.
• Mocks:
  • mock/server/artifact.go added to support API tests.

Example

• Standalone step

  - name: collect-artifacts
    image: golang:1.20
    ruleset:
      status: [failure, success]
    artifacts:
      paths:
        - "test-results/*.xml"
        - "cypress/screenshots/**/*.png"
        - "cypress/videos/**/*.mp4"

[codecov]

Codecov Report

❌ Patch coverage is 42.74809% with 525 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.80%. Comparing base (0ea4d84) to head (1b076c9).

Files with missing lines	Patch %	Lines
api/storage/storage.go	0.00%	96 Missing ⚠️
api/types/artifact.go	0.00%	66 Missing ⚠️
api/admin/storage.go	0.00%	48 Missing ⚠️
database/types/artifact.go	0.00%	47 Missing ⚠️
router/middleware/artifact/artifact.go	0.00%	29 Missing ⚠️
api/artifact/create.go	0.00%	20 Missing ⚠️
api/artifact/get.go	0.00%	15 Missing ⚠️
database/artifact/get_build.go	0.00%	14 Missing ⚠️
cmd/vela-server/metadata.go	0.00%	13 Missing ⚠️
storage/minio/list_objects.go	65.71%	12 Missing ⚠️
... and 33 more

❌ Your project check has failed because the head coverage (57.80%) is below the target coverage (90.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1385      +/-   ##
==========================================
- Coverage   58.36%   57.80%   -0.56%     
==========================================
  Files         645      688      +43     
  Lines       24864    25779     +915     
==========================================
+ Hits        14511    14901     +390     
- Misses       9681    10186     +505     
- Partials      672      692      +20     

Files with missing lines	Coverage Δ
compiler/native/validate.go	100.00% <100.00%> (ø)
compiler/types/pipeline/artifact.go	100.00% <100.00%> (ø)
compiler/types/pipeline/container.go	82.81% <100.00%> (+0.13%)	⬆️
compiler/types/yaml/secret.go	69.76% <ø> (ø)
compiler/types/yaml/step.go	95.65% <100.00%> (+0.09%)	⬆️
database/artifact/count.go	100.00% <100.00%> (ø)
database/artifact/delete.go	100.00% <100.00%> (ø)
database/artifact/index.go	100.00% <100.00%> (ø)
database/artifact/table.go	100.00% <100.00%> (ø)
database/database.go	67.44% <ø> (ø)
... and 51 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.