VMWare vms randomly losing connectivity

[mkolus]

Hello,

First things first:

• OS: Windows 10 Pro
• VMWare Workstation 15
• GNS3 Version: 2.2.7 + GNS VM 2.2.7 running on said hypervisor, freshly reinstalled.
• Windows Server VMWare VM [172.10.10.10] also on VMWare Workstation.
• FortiGate appliance [172.20.10.1] running on the GNS3 VM.
• Both devices are connected to a GNS3 ethernet switch and it doesn't matter that if it runs on my local server or the GNS3 VM.

Let's say that I connect the cable from the Windows Server to the ethernet switch and it works for a while. Then, out of nowhere (I don't touch anything) it losses the connection.

I did some digging and noticed that GNS3 changed the network adapter to vmnet10, as espected and If I sniff traffic while it's working I see bidirectional traffic on vmnet10 and the ubridge encapsulated packets.

When it stops working, I see only the Windows Server ARP requests in vmnet10, while in the ubridge tunnel I see both arp request and replies... as if it somehow stopped injecting packets in vmnet10. The bridge is still listed, though:

bridge list
100-OK
bridge list
101 ethernet0.vnet (NIOs = 2)
100-OK
bridge show ethernet0.vnet
101 bridge 'ethernet0.vnet' is running
101 Source NIO: \Device\NPF_{320C4731-9E0D-4777-AF47-D696FB43FB55}
101 Destination NIO: 10003:192.168.11.1:10004
100-OK
bridge get_stats ethernet0.vnet
101 Source NIO: IN: 100 packets (9502 bytes) OUT: 100 packets (0 bytes)
101 Destination NIO: IN: 100 packets (20250 bytes) OUT: 100 packets (9502 bytes)
100-OK
bridge list
101 ethernet0.vnet (NIOs = 2)
100-OK
bridge show ethernet0.vnet
101 bridge 'ethernet0.vnet' is running
101 Source NIO: \Device\NPF_{320C4731-9E0D-4777-AF47-D696FB43FB55}
101 Destination NIO: 10003:192.168.11.1:10004
100-OK
bridge get_stats ethernet0.vnet
101 Source NIO: IN: 152 packets (13334 bytes) OUT: 138 packets (0 bytes)
101 Destination NIO: IN: 139 packets (24530 bytes) OUT: 152 packets (13334 bytes)
100-OK
bridge get_stats ethernet0.vnet
101 Source NIO: IN: 161 packets (13712 bytes) OUT: 138 packets (0 bytes)
101 Destination NIO: IN: 139 packets (24530 bytes) OUT: 161 packets (13712 bytes)
100-OK
bridge get_stats ethernet0.vnet
101 Source NIO: IN: 168 packets (14006 bytes) OUT: 138 packets (0 bytes)
101 Destination NIO: IN: 139 packets (24530 bytes) OUT: 168 packets (14006 bytes)
100-OK
bridge get_stats ethernet0.vnet
101 Source NIO: IN: 173 packets (14216 bytes) OUT: 138 packets (0 bytes)
101 Destination NIO: IN: 139 packets (24530 bytes) OUT: 173 packets (14216 bytes)
100-OK
bridge get_stats ethernet0.vnet
101 Source NIO: IN: 175 packets (14300 bytes) OUT: 138 packets (0 bytes)
101 Destination NIO: IN: 139 packets (24530 bytes) OUT: 175 packets (14300 bytes)
100-OK
bridge get_stats ethernet0.vnet
101 Source NIO: IN: 179 packets (14468 bytes) OUT: 138 packets (0 bytes)
101 Destination NIO: IN: 139 packets (24530 bytes) OUT: 179 packets (14468 bytes)
100-OK

I repeated the stats command when I lost connectivity.

FWIW, I've attached the packet captures. Feel free to ask if you need anything else, I can reproduce the problem in minutes.

Max




captures.zip

[mkolus]

I forgot something that might be important: I'm using npcap 0.9991. winpcap is not installed in my machine.

DiagReport-20200504-213723.txt

[spikefishjohn]

Not sure if this is related but I just made a new GNS3 server on raw iron (no gns3 vm) ubuntu 20.04 and found a lot of stability issues where GNS3 would report timeouts to localhost when editing switch configs. We would also see VMs become unassailable through a bridge interface (br0) that was bridged to a physical nic. This was on ubuntu 20.04 so everything was built from pip3 and source by hand. ubridge 0.9.19 was installed (from git pull). This was strange because we have another box running 20.04 without issue. After comparing all installed packages I found out the 20.04 box that was having issues was running ubridge 0.19 (latest release).

Stability seems to have been resolved by downgrading ubridge to 0.9.16.

FYI we installed the ubridge from ubuntu 19.04 (disco).

[grossmj]

Actually the latest version is 0.9.18 which can be installed using our PPA:

https://launchpad.net/~gns3/+archive/ubuntu/ppa/+packages?field.name_filter=ubridge&field.status_filter=published&field.series_filter=

sudo add-apt-repository ppa:gns3/ppa
sudo apt-get update
sudo apt-get install ubridge

I found out the 20.04 box that wasn't having issues was running ubridge 0.19 (latest release).

Do you mean the version that is having issue is uBridge version 0.19?

[spikefishjohn]

I just made some updates to make my post a little more clear. I don't have access to that box until monday. I'll check the local repo to see what I have. Now that I think about it I didn't ref a tag when I cloned so it must have been master. I want to say the box was built about 2 weeks ago.

[grossmj]

@spikefishjohn no worries, please let me know what you find 👍

[mkolus]

Actually the latest version is 0.9.18 which can be installed using our PPA:

grossmj, I'm running GNS3 on Windows 10, I'll try to compile ubridge tomorrow and tell you if it worked.

thanks

[spikefishjohn]

So it was for sure head or master.. whatever the git term is. so up to april 28th commit. ubridge labeled as version 0.9.19.

Also, as luck with have it problem seems to have come back. Are you ok with update on this bug report or should I make a new one?

I'm not %100 its related to ubridge.

Are there limits to how big a switch can be? We have a 64 port GNS3 switch that everything is coming into with every 4 ports being in a different vlan. Starting to wonder if its related to dynamips as when I hit stop it never stopped all the dynamips processes.

[spikefishjohn]

I found out my dynamips install was incredibly old for some reason. Maybe the version that comes with ubuntu? Not sure but i took the one from my disco install seems to be running smother.

I've also noticed my notes on how to install this is terrible. I'll need to gns3 a gns3 server to make sure i'm doing this right.

Sorry for the smurfing line noise on the bug report! night all!

[grossmj]

@mkolus

I forgot something that might be important: I'm using npcap 0.9991. winpcap is not installed in my machine.

Yes, this is important. We have had reports that only installing npcap leads to some issues. I would recommend to first install npcap (without winpcap compatibility mode) and then winpcap.

[grossmj]

@spikefishjohn

Also, as luck with have it problem seems to have come back. Are you ok with update on this bug report or should I make a new one?

I think we have different problems here. Please create a new GitHub issue to investigate 👍