"correct 2-sided splitting implementation"

[yiwenzhang92]

Making 2-sided splitting work in the correct way might seem harder than it sounds. When the application is using 2-sided verbs in an aggressive way (e.g., posting and polling in separate threads; plus doing sending and receiving at the same time), careful design is required. Why?
(1) How to make sure the flow control message does not collide when the application is doing posting and receiving at the same time. One possible solution: using another channel (which means one extra pair of split QP) mainly for flow control messages. When sending lots of 2-sided messages, extra number of QPs can cause overhead when we start many connections. This is the current implementation. We do not use 2-sided very often so this is fine.
(2) How to maintain the correct order of data being transferred? To make it simple, this question is equivalent to "How can the remote side know he's going to receive some split chunks, and how to make sure the user can only get the work completion when the last piece of data finishes transferring?" I couldn't think of a clean/elegant way to solve this issue.
Whether people actually use SEND/RECV to transfer a large amount of data (instead of using WRITE/WRITE_IMM) is another question to consider.