Difference between IORING_OP_SENDMSG_ZC and MSG_ZEROCOPY

[davidzeng0]

What's the difference between IORING_OP_SENDMSG_ZC and MSG_ZEROCOPY?

[metze-samba]

It just provides a much saner api, as the notification to allow the buffers to be reused comes as CQE with IORING_CQE_F_NOTIF.
You don't need to use poll for EPOLLERR and use recvmsg(MSG_ERRQUEUE) to get the result

[davidzeng0]

Interesting, I did look a bit at the source code behind IORING_OP_SENDMSG_ZC and it didn't make sense to me why there had to be two CQEs posted for a single send. I never looked into the original MSG_ZEROCOPY documentation and did not know about this error queue.

[isilence]

Right, they share the same idea, but

1. IORING_OP_SENDMSG_ZC is io_uring specific and integrated so all completions are in the io_uring CQ (no more error queues),
2. The buffer notifications are per-request instead of sequence based
3. There are additional features like IORING_RECVSEND_FIXED_BUF saving some cycles and it'll get more
4. And the performance characteristics are different, thanks to the saner API and all other points above.

There is some documentation (need to be extended though)
https://github.com/axboe/liburing/blob/master/man/io_uring_enter.2#L1097

[optimistyzy]

Right, they share the same idea, but

1. IORING_OP_SENDMSG_ZC is io_uring specific and integrated so all completions are in the io_uring CQ (no more error queues),
2. The buffer notifications are per-request instead of sequence based
3. There are additional features like IORING_RECVSEND_FIXED_BUF saving some cycles and it'll get more
4. And the performance characteristics are different, thanks to the saner API and all other points above.

There is some documentation (need to be extended though) https://github.com/axboe/liburing/blob/master/man/io_uring_enter.2#L1097

@isilence Two questions from me:
1 Which kernel is available for this feature? Currently, I have a kernel with 6.0.0-060000-generic, then use io_uring_register_probe to probe the SENDMSG_ZC op in liburing, but it is not avaible.
2 If I do not use the fixed buffer(s), will there still be performance improvement?

Thanks.

[metze-samba]

Regarding 1: SEND_ZC is in 6.0, while SENDMSG_ZC will be in 6.1
Regarding 2: I see improvements when using SENDMSG_ZC in samba without fixed buffers, but the buffers I use are up to 8 Mbytes,
Also note IORING_RECVSEND_FIXED_BUF is only available for SEND_ZC, but not SENDMSG_ZC

[isilence]

send_zc and sendmsg_zc indeed came in 6.0 and 6.1 respectively. Do you have an example program?

[isilence]

Elaborating on 2, both fixed and non-fixed avoid copies if conditions are right, that's the main win. Fixed buffers have some advantages, but what you'll really see will depend on how much of protocol / app / etc. overhead you have to offset not having those advantages. The per-request IO size will be an important factor here.

As for how exactly fixed buffers help: they 1) eliminate page referencing / pinning from the IO path, 2) save on page table traversals (resolving userspace addresses to physical pages) and 3) have small bonuses for internal accounting, which e.g. uses atomics.

[optimistyzy]

@isilence @metze-samba Thanks for your info. For 1, it seems that I need to upgrade my kernel version again; For 2, actually I hope to use TCP with SENDMSG_ZC. Since I use O_NONBLOCK, there will be partial read/write cases (My own program logic will handle this). So if I use
SENDMSG_ZC, what kinds of corner cases I need to take care.

Currently, I see performance drop while doing the test compared with MSG_ZEROCOPY approaches. I deployed my test in two machines. With the application no change in the initiator side, and I use MSG_ZEROCOPY approach VS SENDMSG_ZC approach.

I found there will be two CQEs generated for one sendmsg operations. For example, if I use sendmsg_zc to write data with 8KB. The first cqe: with the real length completed (e.g., 5000 bytes completed with res=5000, and cqe flags = IORING_CQE_F_MORE), the second cqe (with res =0, cqe flags = IORING_CQE_F_NOTIF). Thus it means that two CQEs are needed, then I can issue the next write operations. For TCP streams, there will be partial writes (5000/8000 bytes), so you cannot call immediately write again if the second CQE is not finished (to release buf), because you need to reconstruct your iovs inside sendmsg structure.

Thanks.