Implement end-point to replay unhandled and undispatched events

[arunbakt]

Context:

Domain events are generated in field-data service that are dispatched to rabbitmq messaging service. Similarly events are received and handled by field-data service that originate from admin-panel service. The current event that are generated and received by field-data service are RawCaptureCreated and VerifyCaptureProcessed respectively.

There are two cases where some of the messages generated and received may not end up in a terminal state. (terminal state is the final state indicating the messages are processed):

1. The events generated (events originating from field-data service) could be end-up undispatched to the queue in cases where some error or down-time in messaging service leaves them in raised state instead of transitioning it to sent state. On successfully dispatching, the status moves sent.

2. The events that field-data is subscribed to, are received and handled. But in cases where there is a business case to defer the messages to be processed later, the received images stays in the received state.

We need a mechanism to read all the events in domain_event table that are in received and raised state to be processed in the background and move them to their final state.

Requirement:

Develop an api end-point that retrieves all events that are in the status raised and dispatch them to the appropriate rabbitmq publication (please see messaging implementation here. On successful dispatch mark the domain_event entry as sent.

Similarly, all events that are in received status needs to be retrieved and a appropriate event handler needs to be invoked to process the event. On successful processing, mark the event as handled.

It would be nice to have the ability to keep count of how many times an event is attempted to be dispatched or processed. If the attempts exceed a set threshold, mark the event as failed and do not attempt to reprocess. Another aspect is to have the ability to schedule the time at which to reprocess the event. This could be done by adding a column reprocess_at to the domain_event table that takes a future timestamp. This reprocess_at could default to now() for most messages but could come handy when we know some messages are to be reprocessed in 6 hours time or in a day etc. On each invocation of the replay endpoint, the sql query could retrieve all rows that are below or equal to the reprocess_at value and in status received/raised and process them to move them to the state of sent and handled.