Retry Budget - Can you please explain?

[kitkars]

I have a simple app which is designed to return 500 error to test linkerd retry. This is a GET request. not POST
If I access the app with below retry budget - It retires 10 times. Why? (from where it is coming?)
I thought Retry ratio - for every 10 calls - should be 1 retry? (what does it even mean? for every 10 failures will we retry 1?)

spec:
  retryBudget:
    retryRatio: 0.1
    minRetriesPerSecond: 1
    ttl: 10s
  routes:
  - condition:
      method: GET
      pathRegex: /app
    name: GET /app
    isRetryable: true

[kleimkuhler]

Hi @kitkars. I may not be correctly understanding your question here. Are you asking about how retry budgets work?

It seems like you are correctly understanding it already; Linkerd uses retry budgets, which means that it will always keep a certain ratio of requests and retries.

In this case the retry ratio is 0.1 (10%). So for every 10 requests we will retry 1 if any of those requests fail.

Does this answer your question?

[kitkars]

@kleimkuhler

I am feeling like it does not work as you might have expected.

I designed a simple app which is supposed to fail with 500 error every time. (PLEASE DO NOT USE random.)

The problem is - for the above setup it retries 10 times.

I tried to access the app. /app just one time.
Linkerd keeps retrying 10 times. Then I finally got the response saying that it failed.

I feel like it is more of a bug.

[kleimkuhler]

@kitkars So looking at minRetriesPerSecond, this tells us that at a minimum, we will retry 1 request every second. The ttl tells us that we'll consider retrying this request for 10 seconds (10s).

Therefore, if you send only 1 request and the service fails every time, then it will be retried 1 time for 10 seconds--the 10 times that you are seeing.

If you change ttl to 1s and you should see the number of retries change with that.

[kleimkuhler]

@kitkars So looking at minRetriesPerSecond, this tells us that at a minimum, we will retry 1 request every second. The ttl tells us that we'll consider retrying this request for 10 seconds (10s).

Therefore, if you send only 1 request and the service fails every time, then it will be retried 1 time for 10 seconds--the 10 times that you are seeing.

If you change ttl to 1s and you should see the number of retries change with that.

The retryRatio is the percentage of requests that can be retried and is in addition to the above.

[kitkars]

@kleimkuhler

It sounds like the whole retry process should happen in 10 seconds window with some delay. That is 1 retry / second as it is min retry. After 10 seconds, 10 retries - I should get the final response. But all the 10 retries get completed pretty quickly within 2/3 seconds.
If you think it should not be the case, I can give a reproducible yaml which you could use to replicate. I need sometime for that.

If you think this behavior is right, then Is there any way to add a delay between retries?

[kleimkuhler]

@kitkars Interesting. That does sound like an issue if you are seeing all retries happen within 2/3 seconds.

If you can provide a manifest to reproduce this issue with I would definitely be interested in trying it out and digging in a little more!

[kitkars]

Ok Sir. Sure.

[adleong]

I think there's a little bit of confusion over exactly what the ttl means. The ttl is the window over which the retry budget is calculated. So in your example above, your retry budget allows 1 retry per second plus 10% of the requests made. The ttl is 10 seconds which means that to compute the retry budget, it will look at the past 10 seconds. Assuming no requests have been made in that period, this would be 10 seconds * 1 retry per second = 10 retries allowed in this period.

Assuming that your service responds very quickly, Linkerd will do these 10 retries quickly, consuming the entire retry budget and then returning the error. If you make a second request, no retry budget will be available until 10 seconds later and no further retries will be attempted until that time. This is all by design.

[cpretzer]

@kitkars does the extra info around ttl help?

[kitkars]

@adleong @cpretzer Kind of helps. Thanks.

Question 1:
Is there any way to add a delay between retry requests instead of bombarding 10 times?

Question 2:
If you make a second request, no retry budget will be available until 10 seconds later and no further retries will be attempted until that time. This is all by design.
Please do not mistake me with my question. Do you really think the retry budget design makes sense? It seems to be little bit of confusing. So if one failed request used all the retry budget, the second failed request will not be retried?

[adleong]

The backoff between reties is not configurable.

The design of retries is intended to address random errors. The idea here is that if you have a request which fails, it is retried so that it might succeed the second time (or the third time, etc.). In this scenario, a single request will only need to be retried a few times before it succeeds and it will not consume the entire retry budget. If you have a request which always fails, no amount of retries will help and my recommendation is that enabling retries is not appropriate for that scenario.