fix: implement backoff retry policy for websocket handler

[satyajeetkolhapure]

Web socket handler receives closed connection event more frequently than expected from infura server. So to make ganache more robust a backoff retry mechanism is implemented.

Retry configuration - 3 retries with 2 seconds initial retry and with exponential attempts followed.

When connection is closed, in flight requests are stored temporarily and processed once connection retry attempt succeeds.

Note: Once connection is successful, the retry counter is reset to initial counter.

[tenthirtyone]

There is also a inFlightRequests that may need care/attention here. I am not sure if the desired behavior would be to kill/cancel them or attempt to rebroadcast if/once the socket reconnects and if that's possible. I don't know this part of code. I would assume the latter but @davidmurdoch will need to clarify

[satyajeetkolhapure]

Hi @davidmurdoch : Will this change have any impact on inFlightRequests?

[satyajeetkolhapure]

@tenthirtyone , @davidmurdoch : In flight requests are now stored when connection is closed and processed once connection retry attempt succeeds

[cds-amal]

Not a review, I just had a question.

[MicaiahReid]

Would it be possible to get tests added for this change?

[satyajeetkolhapure]

Would it be possible to get tests added for this change?
Yes. I am going to add few tests to check InFlightRequests too.