Cannot get Operational Webhooks working.

[boothj5]

I am running Svix locally, using docker-compose as per the README. Everything has been working fine, and I have webhooks being sent from the Svix Server Container to listeners on the host. I used svix-server jwt generate for this.

I am now trying to get the Operational Webhooks working. I have followed the guide on the README, generated the token for org_00000000000SvixManagement00 and used it to create the org_23rb8YdGqMT0qIzpgGwdXfHirMu application, and the endpoint for that application to receive operational webhooks.

curl -X 'POST' \
  'http://localhost:8071/api/v1/app/' \
  -H 'Authorization: Bearer <token>' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
        "name": "Operational webhook for default org",
        "uid": "org_23rb8YdGqMT0qIzpgGwdXfHirMu"
    }'

curl -X 'POST' \
  'http://localhost:8071/api/v1/app/org_23rb8YdGqMT0qIzpgGwdXfHirMu/endpoint/' \
  -H 'Authorization: Bearer <token>' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
        "url": "http://host.docker.internal:8777/operations",
        "filterTypes": [
          "endpoint.created",
          "endpoint.deleted",
          "endpoint.disabled",
          "endpoint.updated",
          "message.attempt.exhausted",
          "message.attempt.failing"
        ]
    }'

However when I create a new endpoint in the org_23rb8YdGqMT0qIzpgGwdXfHirMu account, I see the following in the container logs:

WARN svix_server::core::operational_webhooks: Operational webhooks are enabled but no listener set for org_23rb8YdGqMT0qIzpgGwdXfHirMu

And am not seeing any webhooks being sent to the endpoint.

[tasn]

Do you see the 404 trying to send the operational webhooks in the server logs? What did you set for the operational webhooks URL?

[boothj5]

I've put the logs in debug mode, and this is what I see when creating a new endpoint:

server-backend-1    | 2023-11-17T15:18:48.565752Z DEBUG HTTP request{http.client_ip= http.flavor=1.1 http.host=localhost:8071 http.method=POST http.route=/api/v1/app/:app_id/endpoint/ http.scheme=HTTP http.target=/app/app_2YJC8MRxvAVj9bWAvqZvJXOXcVX/endpoint/ http.user_agent=curl/8.1.2 otel.kind="server" request_id=2YJCWis1rsHo3ZgWglnOeYXa2fk trace_id=}: tower_http::trace::on_request: started processing request
server-backend-1    | 2023-11-17T15:18:48.603883Z DEBUG HTTP request{http.client_ip= http.flavor=1.1 http.host=localhost:8071 http.method=POST http.route=/api/v1/app/:app_id/endpoint/ http.scheme=HTTP http.target=/app/app_2YJC8MRxvAVj9bWAvqZvJXOXcVX/endpoint/ http.user_agent=curl/8.1.2 otel.kind="server" request_id=2YJCWis1rsHo3ZgWglnOeYXa2fk trace_id= org_id="org_23rb8YdGqMT0qIzpgGwdXfHirMu" http.status_code=201 otel.status_code="OK"}: svix_server::core::otel_spans: finished processing request latency=38 ms status=201
server-backend-1    | 2023-11-17T15:18:48.607536Z  WARN svix_server::core::operational_webhooks: Operational webhooks are enabled but no listener set for org_23rb8YdGqMT0qIzpgGwdXfHirMu

For the host I have tried both http://127.0.0.1:8071/ and http://localhost:8071/ in the docker-compose.yml services.backend.environment.SVIX_OPERATIONAL_WEBHOOK_ADDRESS

I also have these settings:

SVIX_WHITELIST_SUBNETS: "[ 0.0.0.0/0 ]"
SVIX_DANGEROUS_DISABLE_TLS_VERIFICATION: "true"
SVIX_CACHE_TYPE: "memory"
SVIX_QUEUE_TYPE: "memory"

[tasn]

The "Operational webhooks are enabled but..." line means that it tried to hit the svix server (your local instance) and wasn't able to find an app for your org id so it 404ed. This 404 should show in the logs. If it doesn't it may mean that it's hitting the wrong server for the operational webhooks (and that's why it 404s).

Are you running one svix instance? Or one server, one worker, etc?

[boothj5]

I'm running Svix by docker-compose up in the server folder of the repo, so I am assuming there's just one instance.

I'm also assuming that when I create a new app and endpoint using the JWT from: svix-server jwt generate, this means that the org id I should be using for the operations application is the default one org_23rb8YdGqMT0qIzpgGwdXfHirMu

[tasn]

These are all fair assumptions. Would you mind DMing me up at svix.com/slack/ and we'll get this sorted more interactively?

[boothj5]

A PR has now been created to update the README: #1137

And an issue has been raised for potential improvements: #1138