RoadRunner Temporal worker in version >2.8.2 does not start when ran as a GitLab service in CI pipeline

[benkelukas]

Hello, I have a very interesting problem in GitLab CI pipeline.

I run a Temporal worker as a service in GitLab CI pipeline for integration tests, so that we can test the production image, instance of Temporalite also runs as a service in GitLab CI under the hostname temporalite.

Up until RoadRunner version 2.8.2 everything was working OK, any version above this, the worker does not start, I was able to get logs from that service, and it tries to connect to temporal on 127.0.0.1:7233 instead of the configured temporalite:7233. This is configured via env variable, but does not work even when hardcoding it into .rr.yaml or overriding the service's entrypoint and using --override temporal.address=temporalite:7233

If I run the built image locally in a Kube cluster, it works. When I use it as an image for GitLab's job, it works, it just does not work when used as a GitLab service and only versions above 2.8.2

I've dumped the env variables from the service's container, everything is there correctly.

I've tried overriding the workdir, also does not help.

This is my Dockerfile for the worker:

FROM production-setup as production

COPY /app /app # 

COPY --from=spiralscout/roadrunner:2.10.7 /usr/bin/rr /usr/local/bin/rr

CMD ["/usr/local/bin/rr", "serve", "-c", "/app/.rr.yaml"]

This is my .rr.yaml:

version: "2.7"

rpc:
  listen: tcp://0.0.0.0:6001

server:
  command: "php worker.php"
  env:
    RR_MODE: http
status:
  address: localhost:2114

temporal:
  address: ${TEMPORAL_CLI_ADDRESS}
  activities:
    command: "php worker.php"
  env:
    RR_MODE: temporal
  metrics:
    # Server metrics address
    #
    # Required for the production. Default: 127.0.0.1:9091, for the metrics 127.0.0.1:9091/metrics
    address: 0.0.0.0:9091
    # Metrics type
    #
    # Default: "summary". Supported values: summary, histogram
    type: "summary"

    # Temporal metrics prefix
    #
    # Default: (empty)
    prefix: "prefix"
  namespace: ${TEMPORAL_NAMESPACE}

This is the relevant portion of .gitlab-ci.yaml:

entitlement-processing-service-integration-test:
  stage: Test (Post-Build)
  image: ${PHP_CI_IMAGE}
  services:
    - name: ${WORKER_IMAGE}
      alias: worker
      entrypoint:
        - /usr/local/bin/rr
      command:
        - serve
        - -c
        - /app/.rr.yaml
        - -w
        - /app
    - name: harbor.futuretech.tools/docker-support/temporalite:latest
      alias: temporalite
      entrypoint:
        - /temporalite
      command:
        - start
        - --ephemeral
        - -n
        - default
        - --ip
        - 0.0.0.0
  variables:
    FF_NETWORK_PER_BUILD: 'true'
    TEMPORAL_CLI_ADDRESS: temporalite:7233
  cache:
    <<: *php-dev-cache
    policy: pull
  needs:
    - build-entitlement-processing-service-worker
    - php-vendor-dev-install
  script:
    - # omitted for brewity

Nothing else changes, configuration is the same, just when I change the RoadRunner version to >2.8.2, things stop working.

I know this is a long shot, but I've been banging my head against the wall for 2 days and got nowhere, so maybe somebody has some idea what might be breaking this setup in RoadRunner >2.8.2?

Thanks :)

[rustatian]

Hey @benkelukas 👋🏻

Up until RoadRunner version 2.8.2 everything was working OK, any version above this, the worker does not start

What do you mean, the worker does not start? Do you mean output from the ./rr workers ?

[rustatian]

I have a guess. It might be in the GitLab services, env variable sets as lowercased. Like temporal_cli_address.

[rustatian]

So TEMPORAL_CLI_ADDRESS becomes -> temporal_cli_address.

[rustatian]

We have this change in 2.8.3: roadrunner-server/server@654cb45
But, I'm trying to understand how this may affect the `GitLab services.

[benkelukas]

Hi @rustatian

turns out, the issue was that the worker service (with roadrunner) started before the Temporalite service, that's why it was refusing connection, I've added a wait script into the worker's entrypoint and it works ok now :)

Not sure what changed after rr 2.8.2 - maybe it starts quicker?

Thanks for helping me with what was not rr's fault after all :)

Cheers

[rustatian]

Not sure what changed after rr 2.8.2 - maybe it starts quicker?

Yeah, we update a pool's allocator; now it allocates workers in parallel. So, if previously we spent 1s per worker to allocate, and for N workers, it was N times 1s; now, in the ideal situation, the whole N workers allocated in 1s.

Thanks for helping me with what was not rr's fault after all :)

My pleasure 🤝