RFC: Stock Docker Compose Integration via harbor.yaml

[kundeng]

Title: RFC: Stock Docker Compose Integration via harbor.yaml

---

Summary

I'd like to propose a feature that enables Harbor to use stock Docker Compose files from upstream projects with zero modifications. This would significantly reduce maintenance burden when integrating complex services like Dify, Langflow, or Lobe-Chat that have their own multi-service compose files.

Problem

Currently, when adding a new service to Harbor, we need to manually rewrite the upstream's Docker Compose file to:

• Rename services to avoid conflicts (e.g., api → dify-api)
• Update container_name with Harbor's prefix convention
• Fix all depends_on and network_mode: service:X references
• Rename volumes to avoid conflicts
• Add harbor-network to all services
• Inject Harbor's env files

This is error-prone, time-consuming, and creates a maintenance burden whenever upstream updates their compose file.

Proposed Solution

Introduce a harbor.yaml configuration file per service that declares transformation rules:

# {service}/harbor.yaml
upstream:
  # Path to stock compose file (can be a git submodule)
  source: ./upstream/docker/docker-compose.yaml
  
  # Prefix for all service names (api -> dify2-api)
  prefix: dify2
  
  # Services to exclude (e.g., nginx when Harbor handles reverse proxy)
  exclude:
    - nginx
    - certbot

# Future sections for extensibility
metadata:
  tags: [backend, api]
  wikiUrl: https://github.com/av/harbor/wiki/dify

configs:
  base: ./configs/config.yml
  cross:
    ollama: ./configs/config.ollama.yml

The CLI would automatically transform the stock compose at runtime:

Original	Transformed
Service api	{prefix}-api
container_name: X	${HARBOR_CONTAINER_PREFIX}.{prefix}-{original}
depends_on: [redis]	depends_on: [{prefix}-redis]
network_mode: service:X	network_mode: service:{prefix}-X
Volume mydata	{prefix}-mydata
Networks	Add harbor-network to all services
env_file	Inject .env and override.env

Benefits

1. Zero maintenance - Stock compose files can be git submodules, updated with git pull
2. Backward compatible - Services without harbor.yaml work exactly as before
3. Extensible - The harbor.yaml schema can grow to include service metadata for the Harbor App, config merging declarations, etc.
4. Reduced errors - Automated transformation eliminates manual rewrite mistakes

Implementation Status

I have a working proof-of-concept implementation:

• Branch: feature/upstream-compose-integration (kundeng/harbor)
• Core module: routines/upstream.ts (~390 lines)
• Test service: dify2/ with stock Dify compose file

The implementation integrates into the existing mergeComposeFiles.ts flow - upstream transforms are loaded before regular compose files and merged using the existing deepMerge logic.

Questions for Discussion

1. Does this align with Harbor's design philosophy?
2. Should the harbor.yaml file live in the service directory (proposed) or somewhere else?
3. Any concerns about the transformation rules? Are there edge cases I'm missing?
4. Interest in the future metadata: and configs: sections for Harbor App integration?

Future Direction: Declarative Cross-Service Overlays

Currently, Harbor uses file-naming conventions for cross-service integration:

• compose.x.aider.ollama.yml - applied when both aider AND ollama are running
• compose.litellm.langfuse.postgres.yml - applied when ANY of those services run

This works well but has limitations:

• Discovery requires scanning filenames
• Logic is implicit in naming conventions
• Hard to see all integrations for a service at a glance

The harbor.yaml approach could evolve to make this declarative:

# aider/harbor.yaml
upstream:
  source: ./upstream/docker-compose.yaml
  prefix: aider

# Cross-service overlays - applied conditionally based on active services
overlays:
  # When ollama is also running
  ollama:
    volumes:
      - ./configs/aider.ollama.yml:/root/.aider/ollama.yml
  
  # When litellm is also running  
  litellm:
    volumes:
      - ./configs/aider.litellm.yml:/root/.aider/litellm.yml
    environment:
      - OPENAI_API_BASE=http://litellm:4000
  
  # Platform capabilities (nvidia toolkit detected)
  nvidia:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

# Multi-service conditions (AND logic)
overlays:
  # Applied only when BOTH litellm AND langfuse are running
  [litellm, langfuse]:
    environment:
      - LANGFUSE_HOST=http://langfuse:3000

This would:

1. Consolidate all cross-service logic for a service in one place
2. Make dependencies explicit and discoverable
3. Support complex conditions (AND/OR logic) declaratively
4. Maintain backward compatibility - file-based overlays continue to work

The file-naming convention could eventually become syntactic sugar that generates harbor.yaml entries, or both systems could coexist.

Why Not Native Docker Compose Features?

Docker Compose has profiles and include, but neither can replace Harbor's dynamic composition:

• Profiles are static - you assign services to named profiles at authoring time. Harbor's file-matching is dynamic based on runtime service selection.
• Include lacks conditional logic - there's no include: if: condition syntax.

Harbor's file-matcher implements a rule engine that neither feature can replicate. The harbor.yaml approach formalizes this as a declarative configuration layer.

Next Steps (if accepted)

• Test with more upstream services (Langflow, Flowise, Lobe-Chat)
• Handle Docker Compose profiles: passthrough
• Optional init container generation for config preparation
• Prototype overlays: syntax for cross-service declarations
• Documentation and migration guide

---

Happy to discuss and iterate on this design. Would love to hear thoughts from the community!

[kundeng]

Longer-Term Vision: Service Manifest for Full Lifecycle

This is admittedly ambitious, but if the harbor.yaml pattern proves useful, it could evolve into a service manifest that handles the full operational lifecycle - not just what to run, but how to operate it:

# Future harbor.yaml - full service manifest
upstream:
  source: ./upstream/docker-compose.yaml
  prefix: myservice

overlays:
  ollama: { ... }
  nvidia: { ... }

# Lifecycle hooks (beyond Docker's native post_start/pre_stop)
hooks:
  pre_up:
    - script: ./scripts/check-dependencies.sh
  post_up:
    - script: ./scripts/seed-database.sh
  pre_down:
    - script: ./scripts/backup-data.sh

# Secrets management (SOPS/age, Vault, etc.)
secrets:
  provider: sops
  files:
    - .env.secrets.enc  # decrypted at runtime

# Backup & persistence
backup:
  schedule: "0 2 * * *"
  volumes: [postgres-data, ollama-models]
  destination: s3://harbor-backups/{{date}}

# Multi-environment support
environments:
  dev:
    overlays: [nvidia]
    env_file: .env.dev
  prod:
    overlays: [nvidia, monitoring]
    secrets:
      provider: vault

This would position Harbor as a compose-based orchestration platform - something I've been searching for across many alternatives:

• Dokploy - Clean UI, good Compose support, but limited cross-service orchestration
• Coolify - Feature-rich PaaS, but focused on app deployment not service composition
• CapRover - Solid and customizable, but Compose support is secondary
• Portainer - Great container GUI, but not an orchestration layer
• Dockge - Simple compose management, lacks advanced composition logic

Harbor is the closest I've found to treating Docker Compose as a first-class orchestration primitive rather than just a deployment artifact. The dynamic file-matching, cross-service config merging, and service-aware CLI are exactly the patterns needed for complex multi-service stacks (especially AI services that need to wire together backends, frontends, and satellites).

The harbor.yaml proposal formalizes these patterns. If proven with AI services, it could become a foundation for an API-first, compose-based orchestration system that surpasses the alternatives for complex scenarios.

(This vision may be too ambitious for Harbor's current scope as an LLM toolkit - happy to keep it focused on the immediate upstream: feature if preferred.)

Next Steps (if accepted)

• Test with more upstream services (Langflow, Flowise, Lobe-Chat)
• Handle Docker Compose profiles: passthrough
• Optional init container generation for config preparation
• Prototype overlays: syntax for cross-service declarations
• Documentation and migration guide

---

[av]

Thank you so much for using Harbor and spending the time on the PR and the proposal!

I promise to leave more substantial feedback later, but didn't want to leave you waiting here.

I've been thinking about similar functionality on and off since the @ahundt's proposal about native services. Something similar is definitely wanted. My current ideas revolve around syntax that is a strict superset of Docker Compose and "compiles" to it, including ability to inherit/merge remote configs, better YAML anchors and more. However, it was nowhere near being actionable at this moment in time.

I'd like to know what you think about such an approach to the orchestration layer, if that makes any sense, we can start designing how some of the abstractions would look like.

[kundeng]

---

Disclaimer: I'm not a programmer by trade - more of a data architect. I understand these patterns at a conceptual level, so please correct me if I'm misframing anything technically. I used Claude Opus to help implement and test the code.

---

Thanks for the fast response! I'm excited to explore this further.

I think we're aligned on the need for richer orchestration capabilities. Let me share how I'm thinking about the architecture:

Two distinct layers of desired state:

1. Compose transformation (build-time) - Take stock Compose files, apply overlays, output valid Compose YAML. This is what the current PR does with harbor.yaml.

2. Runtime orchestration (deploy-time) - Things Compose can't express: placement across hosts, secrets lifecycle, backup policies, health remediation, canary rollouts. This would need a daemon/controller and a separate manifest.

I'd resist calling either of these a "compose superset" because:

• The first layer outputs valid Compose (like Kustomize outputs valid K8s manifests)
• The second layer operates on Compose deployments but isn't Compose at all

On using x- extension attributes:

One approach would be embedding orchestration hints directly in Compose files via x-harbor-* attributes. But I'd push back on requiring them in input files, for the same reason Kubernetes chose Kustomize as a separate file:

1. Base manifests stay portable and untouched
2. Overlay logic can be arbitrarily complex without polluting the spec
3. Different overlays for different environments without file duplication

However, x- attributes are valuable in the output Compose file as annotations:

# Final merged docker-compose.yaml (output)
services:
  dify2-api:
    image: langgenius/dify-api:1.11.1
    x-harbor-source: ./upstream/docker-compose.yaml
    x-harbor-overlays-applied: [ollama, nvidia]
    x-harbor-prefix: dify2
    environment:
      - OPENAI_API_BASE=http://ollama:11434/v1  # injected by overlay

This is similar to how Kustomize adds config.kubernetes.io/origin annotations to output manifests. Useful for debugging, tooling, and idempotency checks - without requiring upstream files to be modified.

Conditional overlays in the orchestration layer:

# harbor.yaml - conditions live here, not scattered in compose files
overlays:
  ollama:
    when: [ollama]  # apply when ollama service is active
    services:
      api:
        environment:
          - OPENAI_API_BASE=http://ollama:11434/v1
  
  nvidia:
    when: [nvidia]  # apply when nvidia capability is active
    services:
      api:
        deploy:
          resources:
            reservations:
              devices: [{ driver: nvidia, count: all, capabilities: [gpu] }]

Concrete separation:

# harbor.yaml - Compose transformation (like Kustomize)
upstream:
  source: ./upstream/docker-compose.yaml
  prefix: dify
overlays:
  ollama: { when: [ollama], ... }
# Output: valid docker-compose.yaml with x-harbor-* annotations

# harbor-runtime.yaml - Orchestration state (only with daemon/controller)
placement:
  dify-api: node-gpu-01
secrets:
  provider: sops
  rotate: weekly
backup:
  volumes: [postgres-data]
  schedule: "0 2 * * *"
# Output: controller instructions

This keeps Compose as the portable artifact while allowing arbitrarily rich orchestration logic in a separate layer. The Compose spec can evolve independently - we're not chasing it or extending it. It's similar to how Elestio uses elestio.yaml as a meta-manifest around standard Compose.

Does this framing resonate? Happy to sketch out what the abstraction boundaries might look like.

---

Current PR status:

The feature/upstream-compose-integration branch has a working implementation of the first layer (Compose transformation). Core logic is in routines/upstream.ts (~400 lines) using proper YAML parsing via @std/yaml.

Tested with Dify's stock compose file:

• ✅ Service prefixing, network/volume transformation working
• ✅ Merged compose validates and runs
• ⚠️ Some Dify services need additional env config (storage, database profiles) - that's Dify-specific, not transformation bugs

The PR is ready for review on the transformation layer. The runtime orchestration layer (harbor-runtime.yaml) is future work that would only be needed if Harbor grows a daemon/controller architecture.

---

[av]

Hi, sorry for a long turnaround on this one, I've been thinking about this on and off and only now got a chance to properly review this last message.

Two distinct layers of desired state

I agree with your about build/deploy duality, this new configuration should be able to express runtime changes for a given service.

One existing example of deploy-time configuration is Harbor assembling a service config from individual parts before starting the service, inside of the service's container. Currently its done by entrypoint override and injecting a few custom scripts, by that's less than ideal.

The second layer operates on Compose deployments but isn't Compose at all

This is an interesting point, cause it's a decision fork: whether to keep Harbor contained to Compose, or start adding extras on top. Currently all of the implemented features are still achievable with native compose, so it's possible to completely eject from Harbor into a native setup. With custom runtime component this won't be true anymore.

I don't yet have an opinion about which one is more appropriate, as I see drawbacks in both of the approaches. What is clear is that going beyound "native" compose capabilities should bring a few "killer" features with it, to hlep justify custom runtime dependencies on the host.

Extensions within compose files vs. separate files

This is the area where my opinion drifted a lot since the last discussion. I now completely reconsidered what I want out of this format, based on my past encounters with k8s, HCL and all other attempts to express infrastructure/services with a language. Initially, I started exploring Jsonnet, Dhall, CUE, but very quickly I realised that this is not a correct way to approach the problem: it's bespoke, custom syntax, nice features but not widely adopted - wrong choice overall.

Then, I remembered my (awful) experience with HCL where every language feature is an afterthought, one can't guess a syntax of anything, completely custom data structures that do not behave like one would expect. This contrasted a lot with my Pulumi experience, where instead of trying to re-invent programming language, the configuration was just using one.

So, that's what I'm focusing on right now in terms of an idea - a TypeScript (because we already use Deno for routines) or Python (widely adopted/available) fully typed SDK with good autocomplete, which can build compose setups with the features needed by Harbor.

I've only started considering how it should look and behave, but curious to hear your thoughts on if something like this makes sense or not.