Keep container/-step alive across dags (when referencing as sub-dag)

[plc-dev]

Hello, I just started to play around with dagu, and I am really liking it so far, but I seemingly can't get the following scenario to work.

I would like to create a container or step that specifies spins up a postgres-db, and exposes the ports in the host network. Ideally, this container/step can be included as a sub-dag in other dags.

I've started from the postgres-example given in #1244, which works perfectly.

That example I've changed to the following dag named postgres:

container:
  image: postgres:17
  env:
    - POSTGRES_PASSWORD=secret
    - POSTGRES_DB=appdb
  keepContainer: true
  network: host
  ports:
    - 5432:5432
  startup: entrypoint
  waitFor: running

steps:
  - name: check-ready
    command: pg_isready -U postgres -d appdb
    retryPolicy:
      limit: 10
      intervalSec: 1

From this example and the note on the container-configuration I assume that the container is kept alive atleast until the sub-dag is fully executed. The initial test with the example given in #1244 seems to confirm this.

However, if I call the postgres-dag in another dag - let's call it postgres_create_table - I seemingly can't connect to the postgres-db.
The dag-spec for 'postgres_create_table` is as follows:

params:
  - CREATE_TABLE: "'CREATE TABLE IF NOT EXISTS demo(id INT PRIMARY KEY, name TEXT);'"

env:
  - PGPASSWORD: secret

steps:
  - call: postgres.yaml
    depends: []

  - name: create-db
    executor:
      type: docker
      config:
        image: alpine
        autoRemove: true
    command: sh -c "apk add postgresql17-client --allow-untrusted && psql -U postgres -h localhost -d appdb -c ${CREATE_TABLE}"

The error message is as follows:

* Setting postgresql17 as the default version
psql: error: connection to server at "localhost" (::1), port 5432 failed: Connection refused
	Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
	Is the server running on that host and accepting TCP/IP connections?

If I run netstat instead of the create-db-step, it yields an empty list.

I also tried changing the startup-property of posgtres-dag to keepalive and manually trigger the entrypoint-script and the startup of the postgres-image. This fails either due to inappropriate rights (sudo-user starting postgres) or the postgresql.conf not being available after explicitly changing the user via su -c postgres postgres with the following error message:

postgres: could not access the server configuration file "/var/lib/postgresql/data/postgresql.conf": No such file or directory

Am I making an obvious mistake, or is it simply not possible as of now to keep a container/step alive across dags? Thanks for any response in advance!

[yottahmd]

Hi @plc-dev,
Thank you for the detailed issue report with error messages (and sorry for the late response). Currently, this isn’t supported natively. I’m not entirely sure what the best approach would be, but one possible way is to create a base.yaml file in your Dagu config directory and define the container setup there.

For example, your base.yaml might look like this:

dotenv: /path/to/env
secrets:
  - name: PGPASSWORD
    provider: env
    key: PGPASSWORD

container:
  image: postgres:17
  env:
    - POSTGRES_PASSWORD=$PGPASSWORD
    - POSTGRES_DB=appdb
  keepContainer: true
  network: host
  ports:
    - 5432:5432
  startup: entrypoint
  waitFor: running

Then all your DAGs can inherit the same configuration.
I’m not completely sure if this will work as expected, so any suggestions or design ideas would be very welcome.