Windows - get_stats shows empty (0 bytes) packets out to local interface

[framer99]

I have a version of the infamous GNS3 Windows 10 host Cloud not working. I have a couple details about the issue I haven't seen in my searching though.

It does work for my regular LAN NIC. The router device I'm using (Mikrotik CHR) can DHCP from the LAN in my house and talk to the internet just like any other computer here in another room.

That's the only interface it works with. I've tried TAPs(media set to always connected and app controlled), Loopbacks(created by GNS#'s loopback-manager.cmd), and a couple additional VMNetX's (created via GNS3's vmnet-manager.cmd).

I finally gave in and worked around it as recommended in
#43

• I added a 3rd "Custom" Network Adapter set to VMNet4 int the GNS3 VM via VMWarePlayer.
• This shows up as eth2 in GNS3 VM if you reboot the VM
• Add Cloud device that runs on GNS3 VM
• Make your connection from Cloud eth2 to your router device.

Anyway, while troubleshooting, here's the interesting thing I found:

With the ubridge tunnel going from GNS3 VM (running the router device) to the local GNS3 server (running the Cloud) the get_stats hypervisor command shows packets properly flowing through the UDP tunnel but the "out bytes" count on one side is always 0.

101 4e2a2ee8-10d8-40a5-963b-3f1ef57d9ac1-1 (NIOs = 2)
100-OK
bridge show 4e2a2ee8-10d8-40a5-963b-3f1ef57d9ac1-1
101 bridge '4e2a2ee8-10d8-40a5-963b-3f1ef57d9ac1-1' is running
101 Source NIO: 10033:192.168.80.128:10053
101 Destination NIO: \Device\NPF_{3C5AB807-29F3-451A-B155-4F4CDCC2C11B}
100-OK
bridge get_stats 4e2a2ee8-10d8-40a5-963b-3f1ef57d9ac1-1
101 Source NIO: IN: 167 packets (8550 bytes) OUT: 197 packets (29284 bytes)
101 Destination NIO: IN: 197 packets (29284 bytes) OUT: 167 packets (0 bytes)
100-OK

Not many packets are shown in the captured stats above but I had previously ran pings from each end and let it run over night. Thousands of packets and bytes matched all around except for OUT on the host going to the Loopback device.

Also, sniffing with wireshark supports the idea the bytes output to the host are empty. I can see both directions of traffic in wireshark capturing via GNS3 (right click on link and start capture). But, if I run wireshark and sniff on the actual Loopback adapter on the Host, I see only outgoing packets. No frames ever make it to host.

The stats and sniffing was the same for TAP, loopbacks and VMnet devices I tried. Firewall was completely disabled for domain,public and private. I also added a ubridge.exe exception just to be sure.

Maybe the 'empty" packets output I've added above will help someone figure something out, or point me where to dig deeper.

[grossmj]

Thanks for the details. We are going to investigate it 👍

[framer99]

I looked at this again....

When it "works" (using my local LAN adapter) the virtual router device can DHCP and talk to the internet or any device on my local lan. Those devices can talk to the virtual router device also. The OUT Bytes still shows 0.

But, a new bit of info is that even with the virtual device working to talk to my LAN, I still can't ping or talk to the virtual router device from my desktop.

I switched to a Loopback adapter again, and found I was mistaken in my wireshark sniffing. Sniffing on the actual loopback adapter on the host I CAN see traffic both ways. I see a good ARP request from my desktop and reply from the virtual router device. But, Windows10 on my dekstop never adds the MAC from the reply to the ARP table though, so nothing proceeds past the ARP process.

I also sniffed the ubridge tunnel connection on the VMNet1 adapter of the VM and decoded the UDP payload as ethernet and the ARP conversation looks good.

I built ubridge from source and was going to swap it inplace of the one from my GNS3 install but it seems like that is not the problem.

So, to summarize the status... OUT Bytes still always shows 0, but the traffic is properly tunneled, and wireshark can see it egressed from the tunnel on the Loopback interface. But, Windows internals are either not seeing the ARP reply to add to the ARP table, or it's ignoring it... on purpose I guess?

In the screenshots below, 192.168.77.1 is the desktop/host loopback adapter IP and 77.2 is the IP I have assigned to the virtual router device.

[framer99]

Ahhh, ok I see how i noticed earlier Wireshark captures not seeing 2 way traffic.

If i run my test ping from the virtual router towards the host, I only see ARP requests when sniffing on the hosts loopback adapter. The host never replies.... because it's not seeing the ARP request even though it shows up in the sniff of the loopback adapter. So that is my previously seen "1-Way" traffic issue.

So, it does seem the be an issue with traffic egressing the tunnel on the host not being seen by the host.

Looking at the code I noticed the FreeBSD specific BIOCFEEDBACK? Maybe the latest Windows 10 or NPCAP's winpcap compatibility mode I'm using needs something like that also?

[grossmj]

I know we have similar issues with this before, more about echoed packets though:

GNS3/gns3-gui#2228 (comment)
GNS3/gns3-gui#1092 (comment)

Perhaps in your case the problem comes from the PCAP_OPENFLAG_NOCAPTURE_LOCAL flag in the code?

Have you tried the latest uBridge available there? https://github.com/GNS3/ubridge/releases/tag/v0.9.16

Maybe the latest Windows 10 or NPCAP's winpcap compatibility mode I'm using needs something like that also?

Please feel free to try the latest version of NPCAP, GNS3 should support it. However, I remember we have had issues with packet capture when using NPCAP... maybe they fixed that too?

[framer99]

Thank you for the info, it's very helpful. I feel confident i'm getting closer to the root cause (not "close" but "closer").

• ubridge.exe I've been using reports v0.9.16 (installed GNS3 all-in-one for windows v2.1.20)
• ubridge.exe built from git (reports v0.9.17), same issue.
• removed the PCAP_OPENFLAG_NOCAPTURE_LOCAL flag and recompiled, same issue still exists.

Tonight or tomorrow I will try to sort out building against WinPcap/NPcap , npcap's compatibilty mode, actually having WinPcap installed etc. I'm not up to speed on all that stuff.

I followed the ubridge build instructions for Windows in the README. I installed cygwin's latest gcc-core and make though. The command to install a 2016 era gcc/make package from fruitbat.com seemed oudated and didn't seem to work for me anyway. I used libs and .h files from the WinPcap SDK as directed.

I'd like to try building using headers and libs from NPcap. I'd also like to try to avoid "compatibility mode" if the changes required aren't too extensive. Are both of those things already done and I just haven't figured that out yet?

I noticed(but didn't read too closely yet) your github comments about switching to NPcap 3 years ago and as I recall the GNS3 installer is aware of NPCap. Is the migration to NPcap still in process and WinPcap compatibility is retained?

I saw this recent issue posted in NMAP/Npcap issues:
#1603 Packets injected with send_packet() cannot be received by sockets in the same computer

So maybe it's purely a bug in NPcap, but I'm thinking it's something to do with some incompatibility that pops up in the compatibility mode setups :)