Skip to content

fix: use ephemeral mode in OrchestrationClient to prevent read-only container failures#20905

Open
Br1an67 wants to merge 1 commit intoPrefectHQ:mainfrom
Br1an67:fix/19317-ha-readonly-automation
Open

fix: use ephemeral mode in OrchestrationClient to prevent read-only container failures#20905
Br1an67 wants to merge 1 commit intoPrefectHQ:mainfrom
Br1an67:fix/19317-ha-readonly-automation

Conversation

@Br1an67
Copy link

@Br1an67 Br1an67 commented Mar 1, 2026

Closes #19317

Background services (e.g. actions service) use OrchestrationClient, which calls create_app(). This triggers UI static directory creation via create_ui_static_subpath(), which fails with PermissionError in read-only containers (common in rootless/secure deployments). The exception is swallowed, causing automation actions to silently fail and messages to go to DLQ with no error logs.

Changes

1. Skip UI creation for OrchestrationClient (src/prefect/server/api/clients.py)

Pass ephemeral=True to create_app() in BaseClient.__init__. Background services use in-memory ASGI transport and don't need the UI — ephemeral=True skips UI directory creation, preventing PermissionError in read-only containers.

2. Add error logging for unexpected action failures (src/prefect/server/events/actions.py)

Catch and log unexpected exceptions in the action consumer before re-raising. This provides visibility when actions fail for reasons other than ActionFailed, instead of messages silently going to DLQ.

3. Regression test (tests/server/api/test_clients.py)

Verifies that OrchestrationClient creates the app in ephemeral mode.

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
  • If this pull request adds new functionality, it includes unit tests that cover the changes
  • If this pull request removes docs files, it includes redirect settings in mint.json.
  • If this pull request adds functions or classes, it includes helpful docstrings.

@codspeed-hq
Copy link

codspeed-hq bot commented Mar 1, 2026

Merging this PR will not alter performance

✅ 2 untouched benchmarks


Comparing Br1an67:fix/19317-ha-readonly-automation (f5c8c29) with main (13e96b7)

Open in CodSpeed

…ontainer failures

Closes PrefectHQ#19317

Background services (actions service) use OrchestrationClient, which calls
create_app(). This triggers UI static directory creation, which fails with
PermissionError in read-only containers (common in rootless/secure deployments),
causing automation actions to silently fail and messages to go to DLQ.

Changes:
- Pass ephemeral=True to create_app() in BaseClient, skipping UI directory creation
- Add exception logging in action consumer so unexpected errors are visible
- Add regression test for ephemeral mode usage
@Br1an67 Br1an67 force-pushed the fix/19317-ha-readonly-automation branch from 03a04e6 to f5c8c29 Compare March 1, 2026 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Automations 'run-deployment' action silently fails in HA when UI static dir is read-only

1 participant