error: using NS_MOREFRAG on p2p1 requires netmap buf size >= 3072

[leleobhz]

Hello,

With last 2 days I'm running out with the following issue:

[57950.668281] 518.031582 [ 769] netmap_update_config      configuration changed for eth0: txring 8 x 512, rxring 8 x 512, rxbufsz 2048
[57950.696808] 518.060110 [1869] netmap_interp_ringid      eth0: tx [0,8) rx [0,8) id 0
[57950.713385] 518.076687 [ 722] nm_mem_assign_group       iommu_group 0
[57950.728492] 518.091794 [1362] netmap_config_obj_allocator objsize 1024 clustsize 4096 objects 4
[57950.746238] 518.109540 [1362] netmap_config_obj_allocator objsize 36864 clustsize 36864 objects 1
[57950.764144] 518.127446 [1362] netmap_config_obj_allocator objsize 2048 clustsize 4096 objects 2
[57950.781913] 518.145216 [1469] netmap_finalize_obj_allocator Pre-allocated 25 clusters (4/100KB) for 'netmap_if'
[57950.802524] 518.165825 [1469] netmap_finalize_obj_allocator Pre-allocated 200 clusters (36/7200KB) for 'netmap_ring'
[57950.862227] 518.225527 [1469] netmap_finalize_obj_allocator Pre-allocated 81920 clusters (4/327680KB) for 'netmap_buf'
[57950.882738] 518.246040 [1629] netmap_mem_finalize_all   interfaces 100 KB, rings 7200 KB, buffers 320 MB
[57950.901541] 518.264844 [1632] netmap_mem_finalize_all   Free buffers: 163838
[57950.920775] error: using NS_MOREFRAG on p2p1 requires netmap buf size >= 3072
[57950.939066] 518.302368 [2045] netmap_mem2_deref         active = 1

Using netmap from my proprietary app or with netmap:p2p1 in libpcap patched or even netmap:p2p1/rt does the same.

Can you please check the issue or tell me what I need to do? I saw some patches regarding ixgbe and dont know if I or the devs need to do something.

Thanks!

[vmaffione]

Yes, we are basically trying to add support for jumbo frames.
You have set a large MTU (larger than 3072 bytes, which is the maximum RX size of your NIC), so your application needs to support NS_MOREFRAG, as you can receive (and send) multi-slot frames.
But it also must be the case that netmap buf_size is larger than 3072, otherwise your NIC could cause out-of-bound access to the netmap buffers and so lead to memory corruption.

[vmaffione]

We are actually still working on this, sorry. We should get back to the old behaviour for ixgbe soon, so stay tuned.

[leleobhz]

Hello!

I really want JumboFrames in Netmap, so thumbs up! I've asked our vendor to change this behaviour and pointed them to this issue.

I really glad you attention to the project and Ill keep tuned!

Thanks folks!

[vmaffione]

Thanks, but actually the Jumbo frame issue was more about e1000, e1000e, igb, etc. ixgbe already had support, but we changed the way it works and we are not happy about that. So we are restoring the old behaviour in short.

[leleobhz]

Hello @vmaffione

Its good to know. I use ixgbe in my work but my personal Lab uses e1000e, so its important to know these kind of difference.

If you need some way a place to tests, you can count on me.

Thanks!

[giuseppelettieri]

Hi @leleobhz, may you please test the latest master on ixgbe, to see if the regression is gone?

[leleobhz]

Surelly @giuseppelettieri

Ill post here the results in a bit!

[leleobhz]

Hello @giuseppelettieri

There is the dmesg:

[147518.702467] ixgbe 0000:04:00.0: removed PHC on p2p1
[147518.796680] ixgbe 0000:04:00.1: removed PHC on p2p2
[147519.303090] netmap: unloaded module.
[147519.329123] 086.835867 [3883] netmap_init               run mknod /dev/netmap c 10 57 # returned 0
[147519.346967] netmap: loaded module
[147519.358802] net nmsink0: netmap queues/slots: TX 1/1024, RX 1/1024
[147519.364421] IPv6: ADDRCONF(NETDEV_UP): nmsink0: link is not ready
[147519.394410] Intel(R) 10GbE PCI Express Linux Network Driver - version 5.3.6
[147519.410315] Copyright(c) 1999 - 2018 Intel Corporation.
[147519.444911] ixgbe: Interrupt Mode set to 2
[147519.457468] ixgbe: Direct Cache Access (DCA) set to 1
[147519.470970] ixgbe: Receive-Side Scaling (RSS) set to 8
[147519.484534] ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
[147519.499129] ixgbe: I/O Virtualization (IOV) set to 0
[147519.512596] ixgbe: Interrupt Throttling Rate (ints/sec) set to 64000
[147519.527605] ixgbe: Enabled/Disable FCoE offload Disabled
[147519.541417] ixgbe: 0000:04:00.0: ixgbe_check_options: FCoE Offload feature disabled
[147519.557930] ixgbe: LRO - Large Receive Offload Enabled
[147519.571609] ixgbe: allow_unsupported_sfp Disabled
[147519.726803] ixgbe 0000:04:00.0: PCI Express bandwidth of 32GT/s available
[147519.742402] ixgbe 0000:04:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[147519.758657] ixgbe 0000:04:00.0 eth0: MAC: 2, PHY: 14, SFP+: 3, PBA No: G73131-003
[147519.775033] ixgbe 0000:04:00.0: a0:36:9f:61:d0:ac
[147519.788157] ixgbe 0000:04:00.0 eth0: Enabled Features: RxQ: 8 TxQ: 8 FdirHash DCA 
[147519.806743] ixgbe 0000:04:00.0 eth0: Intel(R) 10 Gigabit Network Connection
[147519.822572] net eth0: netmap queues/slots: TX 8/512, RX 8/512
[147519.857520] ixgbe: Interrupt Mode set to 2
[147519.869953] ixgbe: Direct Cache Access (DCA) set to 1
[147519.883441] ixgbe: Receive-Side Scaling (RSS) set to 8
[147519.896904] ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
[147519.911397] ixgbe: I/O Virtualization (IOV) set to 0
[147519.924679] ixgbe: Interrupt Throttling Rate (ints/sec) set to 64000
[147519.939519] ixgbe: Enabled/Disable FCoE offload Disabled
[147519.953169] ixgbe: 0000:04:00.1: ixgbe_check_options: FCoE Offload feature disabled
[147519.969715] ixgbe: LRO - Large Receive Offload Enabled
[147519.988217] ixgbe: allow_unsupported_sfp Disabled
[147519.993238] ixgbe 0000:04:00.0 p2p1: renamed from eth0
[147520.031998] IPv6: ADDRCONF(NETDEV_UP): p2p1: link is not ready
[147520.080978] ixgbe 0000:04:00.0: registered PHC device on p2p1
[147520.201356] IPv6: ADDRCONF(NETDEV_UP): p2p1: link is not ready
[147520.215548] 8021q: adding VLAN 0 to HW filter on device p2p1
[147520.277598] ixgbe 0000:04:00.0 p2p1: detected SFP+: 3
[147520.424825] ixgbe 0000:04:00.0 p2p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[147520.440963] IPv6: ADDRCONF(NETDEV_CHANGE): p2p1: link becomes ready
[147521.142830] ixgbe 0000:04:00.1: PCI Express bandwidth of 32GT/s available
[147521.158021] ixgbe 0000:04:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[147521.173964] ixgbe 0000:04:00.1 eth0: MAC: 2, PHY: 1, PBA No: G73131-003
[147521.188961] ixgbe 0000:04:00.1: a0:36:9f:61:d0:ae
[147521.201657] ixgbe 0000:04:00.1 eth0: Enabled Features: RxQ: 8 TxQ: 8 FdirHash DCA 
[147521.219879] ixgbe 0000:04:00.1 eth0: Intel(R) 10 Gigabit Network Connection
[147521.235362] net eth0: netmap queues/slots: TX 8/512, RX 8/512
[147521.250905] ixgbe 0000:04:00.0 p2p1: changing MTU from 1500 to 9710
[147521.482470] ixgbe 0000:04:00.1 p2p2: renamed from eth0
[147521.503398] IPv6: ADDRCONF(NETDEV_UP): p2p2: link is not ready
[147521.551840] ixgbe 0000:04:00.0 p2p1: detected SFP+: 3
[147521.565732] ixgbe 0000:04:00.1: registered PHC device on p2p2
[147521.681340] IPv6: ADDRCONF(NETDEV_UP): p2p2: link is not ready
[147521.690065] ixgbe 0000:04:00.0 p2p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[147521.711090] 8021q: adding VLAN 0 to HW filter on device p2p2
[147523.516409] device p2p1 entered promiscuous mode
[147523.784576] device p2p2 entered promiscuous mode
[147523.824928] ixgbe 0000:04:00.0 p2p1: detected SFP+: 3
[147523.968854] ixgbe 0000:04:00.0 p2p1: NIC Link is Up 10 Gbps, Flow Control: None
[147573.960790] 141.467647 [ 873] netmap_get_monitor_na     eth0 not in netmap mode
[147591.985618] 159.492510 [ 769] netmap_update_config      configuration changed for eth0: txring 8 x 512, rxring 8 x 512, rxbufsz 1500
[147592.014050] info: netmap application on p2p1 needs to support NS_MOREFRAG (MTU=9710,netmap_buf_size=2048)
[147592.469959] info: netmap application on p2p1 needs to support NS_MOREFRAG (MTU=9710,netmap_buf_size=2048)
[147592.533832] ixgbe 0000:04:00.0 p2p1: detected SFP+: 3
[147592.805876] ixgbe 0000:04:00.0 p2p1: detected SFP+: 3
[147592.952684] ixgbe 0000:04:00.0 p2p1: NIC Link is Up 10 Gbps, Flow Control: None
[147596.325826] ixgbe 0000:04:00.0 p2p1: detected SFP+: 3
[147596.472695] ixgbe 0000:04:00.0 p2p1: NIC Link is Up 10 Gbps, Flow Control: None

I've reactivated the large mtu again for this testing. The proprietary application runs well (But not tested if they handle properlly the jumbos). Anyways, Ill request this NS_MOREFRAG to application upstream.

A question: If I capture with libpcap from master (Since netmap was merged early 2017, Im using it, as registered in luigirizzo/netmap-libpcap#2 (comment) ) using device netmap:p2p1/rt - this issue will manifest (Will libpcap capture the jumbos if i ask for, as example using tshark -s 9710) ?

Thanks!

[giuseppelettieri]

I don't think NS_MOREFRAG is correctly used by anybody. It was added experimentally some time ago, but never really supported until now. However, you should be able to make existing applications happy by simply increasing the netmap buffer size to match the MTU:

echo 9710 | sudo tee /sys/module/netmap/parameters/buf_size

then, restart all netmap apps.

[leleobhz]

Hello @giuseppelettieri

Thanks for you feedback! Just a double check, so now any application (That is able to handle jumbo frames) can receive the frames if the buf_size is properlly checked, right? From netmap part, there is no change anymore to support this, right?

Thanks!!

[leleobhz]

And a complementary question:

[159377.006016] 944.535390 [1324] netmap_config_obj_allocator XXX aligning object by 18 bytes
[159377.278692] 944.808065 [ 769] netmap_update_config      configuration changed for eth0: txring 8 x 512, rxring 8 x 512, rxbufsz 1500

This rxbufsz indicates a application-side buffer size change or the option below is in production?

# ▶ cat /sys/module/netmap/parameters/buf_size
9710

Thanks!

[vmaffione]

The answer is yes, but only for ixgbe (for now).
Applications that want to send/receive jumbo frames with e1000, e1000e, igb and i40e need to use multiple slots (with NS_MOREFRAG set on all but the last slot) and set buf_size at least to rxbufsz.
No, rxbufsz is a NIC-specific parameter that says what is the size that the NIC expects for each receive buffer.

[vmaffione]

Can we close this?

[leleobhz]

Surelly @vmaffione ! Thank you so much!