EPIC: p2p stack refactor

[evan-forbes]

Currently, the p2p cannot allocate bandwidth dynamically or cache incoming messages to process in parallel.

While the crucial details such as transport have not yet been determined, we know we want these properties.

• Dynamic allocation of bandwidth
  • This means that if there are message to send, and two connected peers have bandwidth available, it is utilized.
• Cache incoming message without blocking
  • Currently, the p2p stack has the potential to block reading from the tcp buffer as each message is processed serially. Instead, each message should avoid blocking to the extent that it can by the raw bytes immediately being passed to a reactor, or better yet a channel.

Options

There are several different optoins we could take.

• We could simply get rid of the send and recieve rate limiters, and port over the reactor refactor or something similar. Continue using tcp and bbr
  • Refactor reactor wiring logic to support parallel processing of messages #1534
  • remove the send and receive rate limiters
• QUIC prototype Prototype using QUIC instead of tcp and proprietary handshakes #1385
• Replace existing redundant congestion control mech with reactor specific limitations #1546

Do determine which stack is ideal, we should first complete:

• Bandwidth utilization experiment #1530

then reevaluate