[Bug] Some transports block on a delayed requeue where not natively supported

[iancooper]

Describe the bug

Our API offers a delayed requeue where it is possible that the error is transient and could be fixed by simply returning to the queue with a delay. There are three responses we have:

1. Native support for requeue with delay; we used this
2. No native support for requeue and we requeue via a timer event that posts after the elapsed time window
3. We block for the delay period then requeue

We know that 3 is never the right solution as this will block the whole pump, and pause message processing, not requeue with a delay. We should fix those transports that have the incorrect behavior.

Further technical details

• The internal message consumer is a good example of how to get this right with a internal Requeue method that is called via a timer if delay is requested, and just passed through if not.

[iancooper]

As an example, we don't currently support SendWithDelay on AWS SNS/SQS but we should be able to do this.

I don't think we want to u delay queue, as this delays all items. We would need to set up a specific delayqueue for time required and poll that queue for items to repost. Using the visibility timeout seems better, but this might not be possible if not being used from a Requeue as you have to read a message to then increase its lock period

[iancooper]

• Add an abstraction for a scheduled job
• Create an in-memory scheduler that relies on Timers and thus will not persist schedules across runs (i.e works for in 3 seconds ok, but not 3 days, and not reliable)
• Add external schedulers to implement such as AWS Scheduler, Quartz, Hangfire, Redis, Postgres etc.