Consider using "ApproximateReceiveCount" attribute to compute number of immediate processing failures for `ErrorContext`

[jdaigle]

Describe the feature.

Is your feature related to a problem? Please describe.

The number of immediate processing failures is tracked per physical endpoint deployed. This means the actual number of immediate processing failures can exceed the limit before delayed processing kicks in. This can also lead to increased cost.

Describe the requested feature

The ApproximateReceiveCount attribute could be used as the number immediate processing failures passed to the ErrorContext.

I think it starts at 1, so subtract 0 (with a floor of 0) to compute the number of immediate processing failures.

This could be an opt-in setting for now, or perhaps the default behavior in a major upgrade.

Describe alternatives you've considered

Additional Context

No response

[danielmarbach]

Here are some preliminary thoughts.

Currently, our implementation tracks the receive count locally at the endpoint level using an LRU cache. However, this approach introduces reliability issues, particularly in scenarios involving endpoint restarts or scaled-out endpoints. Consequently, the local receive count can become inaccurate or skewed, potentially making it less reliable—or more "approximate"—than the service's provided ApproximateReceiveCount.

Given this observation, a logical improvement would be to replace our current LRU-based tracking entirely with the service-provided ApproximateReceiveCount, eliminating the need for it to be an opt-in feature.

However, there are indications from various sources, such as this Medium article by Zac Charles, that ApproximateReceiveCount from the service might occasionally report a lower number than the actual receive count. If this indeed occurs, an enhanced approach would be to retain our local LRU cache but use it alongside the ApproximateReceiveCount from the service. Specifically, we could compare the two values and consistently choose the higher value, updating our local cache accordingly. This approach would improve reliability by ensuring the receive count is always as accurate as possible, though at the expense of continuing to maintain the complexity and potential limitations of the LRU cache (such as running out of storage).

One edge case worth considering is a situation where a message is received through the portal integration while the endpoint is inactive. For instance, suppose the message was processed once by the endpoint, the endpoint then shuts down, and afterward, the same message is accessed via the portal. In this scenario, the service's ApproximateReceiveCount would likely exceed our locally cached count upon endpoint restart, necessitating an update to our cache. Despite the complexity, maintaining the local cache alongside the service-provided ApproximateReceiveCount could offer significant benefits in ensuring overall accuracy.

It's also important to note that the commonly stated limitation around ApproximateReceiveCount—that it should not be used for deduplication—does not directly impact our current use case.

[danielmarbach]

I have added a quick spike to show how it could look like

[danielmarbach]

@jdaigle Feel free to take a look at the PR that also contains some of the rational of the change

[danielmarbach]

7.3.0 has been released containing this improvement