A Doubt Regarding Multipath Usage ( -A option in picoquicdemo)

[Kazedaa]

I am running a perf test on a simple netns topology to test for multipath usage.(commands to reproduce the topology are given below)

With the client at H1 and the server at H2 using the following arguments.
Client
./picoquicdemo -a perf -M -A 10.0.1.1/<ifindex> 10.0.2.2 55000 "*1:5000000:5000000;"
Server
./picoquicdemo -M -a perf -p 55000

I have made sure that the ifindex for the -A option is correct.
I only get multipath usage when I use the default route(through eth2 at 10.0.1.1 in the -A option

if I use the other interface(through eth2 at 10.0.0.1) I get

Which is confusing to me since the alternative option should the "non default" option. But the opposite behaviors is observed.

Furthermore when I use wireshark to capture packets, only one of the interfaces captures them(the default one, in this case eth2: 10.0.1.1) . Which makes me believe that both Path[0] and Path[1] in the terminal output refer to the same path... and that means that multiple paths are'nt used...

Any help to clear out this confusion for me would help a lot!
Thank you

Commands for the netns Topology

ip netns add h1
ip netns add h2
ip netns add r1
ip link add eth0 type veth peer name eth1
ip link add eth2 type veth peer name eth3
ip link add eth4 type veth peer name eth5
ip link set eth0 netns h1
ip link set eth1 netns r1
ip link set eth2 netns h1
ip link set eth3 netns r1
ip link set eth4 netns r1
ip link set eth5 netns h2
ip netns exec h1 ip link set lo up
ip netns exec r1 ip link set lo up
ip netns exec h2 ip link set lo up
ip netns exec h1 ip link set eth0 up
ip netns exec r1 ip link set eth1 up
ip netns exec h1 ip link set eth2 up
ip netns exec r1 ip link set eth3 up
ip netns exec r1 ip link set eth4 up
ip netns exec h2 ip link set eth5 up
ip netns exec h1 ip address add 10.0.0.1/24 dev eth0
ip netns exec r1 ip address add 10.0.0.2/24 dev eth1
ip netns exec h1 ip address add 10.0.1.1/24 dev eth2
ip netns exec r1 ip address add 10.0.1.2/24 dev eth3
ip netns exec r1 ip address add 10.0.2.1/24 dev eth4
ip netns exec h2 ip address add 10.0.2.2/24 dev eth5
ip netns exec h1 ip route add default via 10.0.1.2 dev eth2
ip netns exec h2 ip route add default via 10.0.2.1 dev eth5
ip netns exec r1 sysctl -w net.ipv4.ip_forward=1

[huitema]

I think the issue is with the command:

./picoquicdemo -a perf -M -A 10.0.1.1/<ifindex> 10.0.2.2 55000 "*1:5000000:5000000;" 

What specific value are you using when you "use the other interface(through eth2 at 10.0.0.1)"?

[Kazedaa]

Oh i realise i had made a typo earlier i meant

"eth2 at 10.0.1.1"

( I have edited the query now)

It's the same command, but the -A option has been changed.
./picoquicdemo -a perf -M -A 10.0.0.1/<ifindex> 10.0.2.2 55000 "*1:5000000:5000000;"

Is there a way to find out what Path[0] and Path[1] are? From the output

That might confirm whether both paths are being used.

QLogs and Outputs

The outputs for both experiments show that 2 and 1867 packets are sent on the "non-default"(through eth0 10.0.0.1), but I can't capture any packets through wireshark on that interface.

The server command for both experiments are the same
./picoquicdemo -F ../output/logs.csv -M -a perf -p 55000

10.0.0.1

QLog
./picoquicdemo -b ../output -F ../output/logs.csv -a perf -M -A 10.0.0.1/7 -G reno 10.0.2.2 55000 "*1:5000000:5000000;"

10.0.1.1

QLog
./picoquicdemo -b ../output -F ../output/logs.csv -a perf -M -A 10.0.1.1/9 -G reno 10.0.2.2 55000 "*1:5000000:5000000;"

[huitema]

You did not answer my question. I was asking what was the command line argument used for starting the client. You answered:

It's the same command, but the -A option has been changed.
./picoquicdemo -a perf -M -A 10.0.0.1/<ifindex> 10.0.2.2 55000 "*1:5000000:5000000;" 

That's not a valid entry, because <ifindex> is a place holder for the actual interface index of the link. What exact argument did you use?

[Kazedaa]

oh sorry i missunderstood what you said.
Here is the ipconfig for the namespace

and the command I used is
./picoquicdemo -a perf -M -A 10.0.1.1/9 10.0.2.2 55000 "*1:5000000:5000000;"

[huitema]

I am looking at the trace for your second attempt, to 10.0.1.1/9. They actually look good. The picoquic UI reports sending data on both paths, the qlog trace shows that the path handshake succeeded:

[3067, 1, "transport", "packet_received", { "packet_type": "1RTT", "header": { "packet_size": 1252, "packet_number": 0, "dcid": "0f8e4ea8b8c95e0a" }, "frames": [{ 
    "frame_type": "path_challenge", "path_challenge": "f53ab373e79882c4"}, { 
    "frame_type": "path_response", "path_response": "dbd85993b4156e70"}, { 
    "frame_type": "padding"}]}],

The log also shows some packets being received on the new path, e.g.:

[3318, 1, "transport", "packet_received", { "packet_type": "1RTT", "header": { "packet_size": 55, "packet_number": 1, "dcid": "0f8e4ea8b8c95e0a" }, "frames": [{ 
    "frame_type": "path_ack", "path_id": 0, "ack_delay": 0, "acked_ranges": [[5, 8]], "ect0": 0, "ect1": 9, "ce": 0}, { 
    "frame_type": "path_ack", "path_id": 1, "ack_delay": 0, "acked_ranges": [[0, 10]], "ect0": 0, "ect1": 11, "ce": 0}, { 
    "frame_type": "padding"}]}],

We have established with the failed attempt on 10.0.0.1 that if the interface ID is not set correctly, the packet will not be forwarded, and only one path will be used. We see here a success with 10.0.1.1/9, which implies that routing worked correctly. As far as I can tell, the server is sending packets over 2 paths, and those packets are received properly.

This could very well be an issue with Wireshark. There is a known issue for wireshark and QUIC Multipath. The packets sent on the alternate path are not identified as QUIC, because Wireshark relies on the initial QUIC handshake to identify connections, and that handshake only happens on the "main" path. What Wireshark filters do you use to check for traffic?

[huitema]

That's strange. The log show that the server is sending packets on both routes. You should see the packets from the server on both captures.

People have made similar configurations work before. I suspect that there is an issue in the way the routing is configured.

@clarkzjw would you have time to look at this configuration? You have more experience than me for this kind of simulation!

[clarkzjw]

@Kazedaa Do you mind trying out the topology at this gist https://gist.github.com/clarkzjw/5bd2a41eb9c9236362b7b02f13217737? This gist uses Mininet though, however you should be able to adapt it to ip netns. If I remember correctly, this topology worked for my tests a while ago. （Please ignore some of the comments in that gist though)

[Kazedaa]

@clarkzjw Sure!
Im assuming i host the server on h1 and client on h2?

Oh and also what exactly is the -A option?
is each ip/ifindex the first interface into the alternate path? or should it have every interface along that path?
from client to server

[clarkzjw]

Im assuming i host the server on h1 and client on h2?

yes.

as for the -A option, you can find an exmaple in #1513 (comment)

[Kazedaa]

Thank you @clarkzjw i found out that i had'nt allowed the router to proxy ARP.
I now capture packets on both interfaces.

And as @huitema pointed out

This could very well be an issue with Wireshark. There is a known issue for wireshark and QUIC Multipath. The packets sent on the alternate path are not identified as QUIC, because Wireshark relies on the initial QUIC handshake to identify connections, and that handshake only happens on the "main" path

Wireshark only recognizes the packets as QUIC-IETF on the default route, where the QUIC handshake happened.

Thank you both very much for your time!