EnvoyExtensionPolicy with Wasm on Gateway causes issues when routing inbound requests

[rsaelim]

Description:
I use HTTPRoute and SecurityPolicies to expose api routes. I applied an EnvoyExtensionPolicy with the goal to count requests base on request headers. After I applied it, all requests with matching HTTPRoutes with an open security policy resulted in 503s, and all requests with matching HTTPRoutes with an authz security policy (using extAuthz gRPC) resulted in 403s.

Repro steps:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway-system
spec:
  telemetry:
    accessLog:
      settings:
        - format:
            type: JSON
            json:
              start_time: "%START_TIME(%Y-%m-%dT%H:%M:%S%z)%"
              authority: "%REQ(:AUTHORITY)%"
              bytes_received: "%BYTES_RECEIVED%"
              bytes_sent: "%BYTES_SENT%"
              x_client_id: "%REQ(X-Client-Id)%"
              duration: "%DURATION%"
              method: "%REQ(:METHOD)%"
              path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
              status: "%RESPONSE_CODE%"
              upstream_host: "%UPSTREAM_HOST%"
              user_agent: "%REQ(USER-AGENT)%"
              x_forwarded_for: "%REQ(X-FORWARDED-FOR)%"
              x_request_id: "%REQ(X-REQUEST-ID)%"
              cluster: "%UPSTREAM_CLUSTER%"
          sinks:
            - type: File
              file:
                path: /dev/stdout
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend-open
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /openRoute
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: open-policy
  namespace: envoy-gateway-system
spec:
  cors:
    allowHeaders:
    - x-api-key
    - authorization
    - content-type
    - user-agent
    allowMethods:
    - GET
    - POST
    - PUT
    - PATCH
    - DELETE
    - OPTIONS
    allowOrigins:
    - '*'
    maxAge: 5m0s
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend-open
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: wasm-extension
  namespace: envoy-gateway-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  wasm:
  - name: wasm-filter
    code:
      type: HTTP
      http:
        url: http://hostname:8080/apiCount.wasm

Environment:
Envoy gateway v1.1

Logs:
envoy-envoy-gateway-system-eg-5391c79d-657b89f787-6ppmz envoy {"path":"/openRoute","bytes_received":0,"x_user_id":null,"bytes_sent":0,"authority":"www.example.com","x_client_id":null,"upstream_host":null,"cluster":null,"duration":0,"method":"GET","x_forwarded_for":"xx.xxx.xxx.xxx","start_time":"2024-09-05T15:22:26+0000","user_agent":"axios/1.5.1","status":503,"x_request_id":"d8c5ccb9-742c-426b-a6fb-32a23bdef6a0"}

^Something odd with the log is that a lot of fields that come from request headers are unexpectedly null, so its possible this is not an EG issue but rather the wasm is messing up the request, though my wasm binary is compiled from something very similar if not nearly identical to this: https://github.com/tetratelabs/proxy-wasm-go-sdk/blob/main/examples/metrics/main.go.

[arkodg]

• do you see any error (Accepted: False) in the status of the policies ?
• do requests work w/o the WASM module ?

[rsaelim]

• do you see any error (Accepted: False) in the status of the policies ?
• do requests work w/o the WASM module ?

No, Accepted: True in security policies and envoyextension policy.
Yes, the requests are routed properly without the wasm module added via the envoyextension policy.