Proxy Behavior when Pod is Down

[luqmansen]

Hello, I have a question regarding the proxy behavior when the container that being proxied is down, is there any retry mechanism that prevents requests from getting error 50X? The only thing that I found regarding the possible case is this docs but didn't specify anything about my question.

The scenario that I did was using WRK to test my apps, my apps can't handle the request, hence the container is down and restarted by Kubernetes (but not the pod), this is the result that I got.

Running 5m test @ http://ip-address
  4 threads and 8000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.31s   383.35ms   2.00s    61.21%
    Req/Sec     1.34k   433.86     4.23k    72.39%
  1599212 requests in 5.00m, 14.98GB read
  Socket errors: connect 0, read 0, write 0, timeout 285739
Requests/sec:   5330.07
Transfer/sec:     51.12MB

Notice that there isn't any single 50X error, but only timeout.

What do I expect
At least there is a 50X error because definitely there is a time where all my pod is down. Either my container is restarted fast enough so the traffic is automatically re-routed to the healthy pod, or kubectl provide falsy information

Is this the expected behavior? Thank you in advance

[olix0r]

When your application container is down, linkerd enters "fail-fast" mode after (I think) 3 seconds. That is, we'll buffer requests for up to 3s and then start failing them with a 502. It may be the case that your application is restarting within 3s so the proxy never enters fail-fast?

[luqmansen]

Ohh i see, that's probably why. I think this behavior should have been documented for clarity. Anyway, thank you for the answer

[cpretzer]

Ohh i see, that's probably why. I think this behavior should have been documented for clarity. Anyway, thank you for the answer

Sorry that you ran into this @luqmansen. We'd love your help adding this detail to the docs if you can submit a PR 😄

[luqmansen]

Sure, I'll create a pr soon