Advice on UDP tuning for image data

[chengguizi]

We are experimenting eCAL cloud pub-sub over direct Gigabit Ethernet connection, and use the default behaviour which is UDP.

https://eclipse-ecal.github.io/ecal/advanced/message_drops.html

From this manual, I understand that we are likely required to tune the OS UDP settings. But could you give examples on what are tunable here?

Setup:

• Ubuntu 20.04 / 22.04
• Communication between arm64 and amd64
• Static IP in the same subnet
• Each ecal message is in capnproto serialisation and size about 1-2MB, 20Hz

Current situration:

• Only able to stream multiple image streams until 25MB/s is hit. Then observable frame drops / lags would happen
• Far from the theoretical 100MB/s, which iperf could run at that speed no issue.

[FlorianReimold]

I would strongly advice you to use TCP for 1-2MB messages. If you only have 1 subscriber over network, you will probably not see any downside of TCP at all. The downside will only come if you have more than 1 subscriber, as each subscriber will get its own data stream and therefore requires extra network bandwith.

You can learn more about the eCAL TCP Layer here:
https://eclipse-ecal.github.io/ecal/advanced/layers/tcp.html

[chengguizi]

Sure! We will switch to that and try. One question. How eCAL's implementation of TCP handles insufficient bandwidth? Say for example we are streaming 50MB/s of image data, but the network is only 30MB/s strong. Congestion would happen. Will the TCP stack queue up indefinitely? or eCAL backend will randomly throw away packets? or just throw a way the oldest / latest?

Is this something tunable?

Some good info is already here https://eclipse-ecal.github.io/ecal/advanced/message_drops.html

[chengguizi]

#1017
Since this is fixed back then, we will test TCP again with Capnproto, compiling from source!

[FlorianReimold]

The TCP protocol brings a congestion control withe the protocol, it is one of the main reasons why somebody would use TCP. So eCAL does not need to implement anything on top of that.

However, you obviously can implement a process that tries to send more data than the phsysical connection can handle. In that case, the eCAL TCP Layer will have a 1-element-queue and replace old messages with newer ones. Old messages are actively dropped, if the transfer via Network hasn't started yet, anyway.

[chengguizi]

I have a specific question on bandwidth_max_udp

Understand the unit is bytes per second, and the actual implementation is here

ecal/ecal/core/src/io/snd_sample.cpp

Line 55 in db15d93

sent_sum = SendSampleBuffer(m_payload.data(), data_size, bandwidth_, std::bind(TransmitToUDP, std::placeholders::_1, std::placeholders::_2, m_udp_sender, m_attr.ipaddr));

ecal/ecal/core/src/io/snd_raw_buffer.cpp

Lines 121 to 126 in db15d93

	if (bandwidth_ > 0)
	{
	send_sleep_us = MSG_BUFFER_SIZE;
	send_sleep_us *= 1000 * 1000;
	send_sleep_us /= bandwidth_;
	}

It seems to throttle each eCAL message by sleeping in between packet.

Question, does that mean this bandwidth is per message / publisher? Not global?

[FlorianReimold]

eCAL can throttle the speed at which it sends out the fragments of a message, yes. It is one possible fix for those cases where you have big messages that result in so many fragments that your UDP buffes are instantly overwhelmed.

That feature comes from a time where eCAL didn't have a TCP Layer. It has a big disadvantage: each eCAL Process limits itself, so as soon as you have 2 eCAL Processes, you will end up with 2 separate heuristics instead of 1 global proper congestion control.

[chengguizi]

Another issue trying to use TCP though:

• As the TCP mode in eCAL relies on hostname resolution
• In our setup, both machines are on static IP, without a router / nameserver
• So there seems no good ways eCAL could establish TCP without tedious /etc/hosts setup?

[FlorianReimold]

Yes, you need some kind of DNS system that resolves a Hostname to an IP. The easiest way on linux is to use /etc/hosts. On Windows, everything should work out of the box.