How to handle partial writes to a network socket?

[redbaron]

If I understand correctly io_uring still preferes to use non blocking sockets , because otherwise it will create io-wq for a blocking send of the whole buffer.

Non blocking send can acccept less bytes than requested, when it happens subsequent write request returns EAGAIN until socket can accept more data. In a non-uring world writing whole buffer to a non blocking socket works like following:

• keep writing until EGAIN or full buffer written
• if EGAIN, poll for POLLOUT readiness, then goto 1

How does it translate to liburing? Do we need to poll or CQE is posted when socket is writable and next write can be made immediately?

[redbaron]

Reading #364 (comment) it seems that "best" mode is O_NOBLOCK and io_uring does the optimal thing:

• doesn't start async thread
• posts CQE when space is available and next write can succeed.

am I right?

[isilence]

Offloading to worker threads (i.e. io-wq) is a slow path, io_uring doesn't do requests this way by default regardless whether it's O_NOBLOCK or not.

I'd recommend, do a normal non-blocking socket and just issue sends. io_uring will internally poll the socket for you if needed. There might be merit in polling if you're streaming (in contrast to more of a ping pong pattern) and running out of space in tx queues.

Also, if you set MSG_WAITALL, for TCP io_uring will try hard to never return short sends, even though it might happen because of allocation failures and/or socket errors. It's done by retrying sending the rest of the data (with polling in between if needed).

[noteflakes]

Also, if you set MSG_WAITALL, for TCP io_uring will try hard to never return short sends, even though it might happen because of allocation failures and/or socket errors. It's done by retrying sending the rest of the data (with polling in between if needed).

Looking at the relevant man pages, I see that MSG_WAITALL only works for recv, not for send. Am I missing something here?

Found it: torvalds/linux@4c3c094

MSG_WAITALL is supported for send in io_uring for SOCK_STREAM and SOCK_SEQPACKET. Not documented though.

[redbaron]

I'd recommend, do a normal non-blocking socket and just issue sends. io_uring will internally poll the socket for you if needed. There might be merit in polling if you're streaming (in contrast to more of a ping pong pattern) and running out of space in tx queues.

assuming TCP connection hangs and no remote peer ACKs can be received. socket sendbuffer is 64kb and it is currently empty, we submit send op for 128kb buffer on a normal socket with no additional flags, what will happen?

1. CQE with 64 kb
2. we submit another send op for remaining 64kb
3. CQE with 0 kb

Or

1. CQE with 64 kb
2. we submit another send op for remaining 64kb
3. io_uring blocks and never returns CQE for the second write

Or

1. io_uring blocks and never returns CQE for the first write, because whole buffer hasn't be sent yet in full

Or

1. io_uring blocks and never returns CQE for the first write, because there is currently no room in socket to make another write

There might be merit in polling if you're streaming (in contrast to more of a ping pong pattern)

In your view, which response size can be seen as streaming?

[isilence]

1. Let's say TCP decides that it can't queue more than 64K, then you will get a CQE with 64KB and the second request will wait until there is more space.

2. If you set MSG_WAITALL, the first request will queue 64KB in the kernel and go waiting until there is more space, so there will be no CQE in your example.

[redbaron]

Thanks for the insight as usual! Sadly I can't mark reply as an answer, would you mind reposting it separately?