🐛 BUG: Two nodes can't connect properly to each other

[QZAiXH]

What version of nebula are you using? (nebula -version)

1.9.5

What operating system are you using?

Windows

Describe the Bug

I have two nodes, A (IP: 172.17.60.163) and B (IP: 172.17.7.238). The operation steps are as follows:

1. ping lighthouse 1 from node A, can connect normally
2. ping lighthouse 1 from node A, it can connect normally. 3. ping lighthouse 1 from node A again, it cannot connect.
3. ping lighthouse 1 from node A again, can't connect normally. 4.
4. Node B can ping lighthouse 1 normally.

Logs from affected hosts

A.log
B.log
l1.err.log
l2.err.log

Config files from affected hosts

Node A configuration

relay:
    relays:
        - 172.17.215.97
        - 172.17.0.1
    am_relay: false
    use_relays: true
tun:
    unsafe_routes: []
    disabled: false
    dev: fsnb0
    drop_local_broadcast: false
    drop_multicast: false
    tx_queue: 500
    mtu: 1300
    routes: []
logging:
    level: info
    format: text
firewall:
    outbound_action: reject
    inbound_action: reject
    conntrack:
        tcp_timeout: 12m
        udp_timeout: 3m
        default_timeout: 10m
    outbound:
        - port: any
          proto: any
          host: any
    inbound:
        - port: any
          proto: any
          host: any
          groups: []
pki:
    ca: xxx
    cert: xxx
    key: xxx
    disconnect_invalid: true
static_host_map:
    172.17.0.1:
        - x.x.x.x:4242
    172.17.215.97:
        - x.x.x.x:4242
lighthouse:
    am_lighthouse: false
    hosts:
        - 172.17.0.1
        - 172.17.215.97
    interval: 60
    local_allow_list:
        interfaces: {}
    advertise_addrs: []
listen:
    host: 0.0.0.0
    port: 0
punchy:
    delay: 1s
    respond_delay: 5s
    punch: true
    respond: true
handshakes:
    trigger_buffer: 64
    try_interval: 100ms
    retries: 10
static_map:
    network: ip4
    cadence: 30s
    lookup_timeout: 250ms
cipher: aes
routines: 1

Node B configuration

relay:
    relays:
        - 172.17.0.1
        - 172.17.215.97
    am_relay: false
    use_relays: true
tun:
    unsafe_routes: []
    disabled: false
    dev: fsnb0
    drop_local_broadcast: false
    drop_multicast: false
    tx_queue: 500
    mtu: 1300
    routes: []
logging:
    level: info
    format: text
firewall:
    outbound_action: reject
    inbound_action: reject
    conntrack:
        tcp_timeout: 12m
        udp_timeout: 3m
        default_timeout: 10m
    outbound:
        - port: any
          proto: any
          host: any
    inbound:
        - port: any
          proto: any
          host: any
          groups: []
pki:
    ca: xxx
    cert: xxx
    key: xxx
    disconnect_invalid: true
static_host_map:
    172.17.0.1:
        - x.x.x.x:4242
    172.17.215.97:
        - x.x.x.x:4242
lighthouse:
    am_lighthouse: false
    hosts:
        - 172.17.0.1
        - 172.17.215.97
    interval: 60
    local_allow_list:
        interfaces: {}
    advertise_addrs: []
listen:
    host: 0.0.0.0
    port: 0
punchy:
    delay: 1s
    respond_delay: 5s
    punch: true
    respond: true
handshakes:
    trigger_buffer: 64
    try_interval: 100ms
    retries: 10
static_map:
    network: ip4
    cadence: 30s
    lookup_timeout: 250ms
cipher: aes
routines: 1

[johnmaguire]

Hi @QZAiXH - I don't understand your steps to reproduce. Number 3 is represented twice. I believe this was a mistake and you intended to include 5 bullet points.

1. From steps 1 to 2 you state there is no change in behavior. Did you take any action between these steps?
2. From steps 2 to 3, the behavior changes. Did you take any action between these steps?
3. Finally, steps 3 and 4 indicate no change - is that correct?
4. You provided logs for a "l2.err.log" which I assume is a second lighthouse. You didn't mention it in the operation steps. Can you explain its relevance?

If you took no actions besides what's listed in these steps, would it be fair to say that your operation steps can be summarized simply as having intermittent connectivity between Node A and Lighthouse 1, while Node B has a stable connection to Lighthouse 1?

[QZAiXH]

I'm sorry, the above content was copied and pasted, and some parts got messed up.😭

Lighthouse 2 is used as a redundant backup, so it wasn't mentioned in the previous description. If needed, I can perform the same tests on it.

The complete reproduction steps are as follows:

1. After Node A starts, it can ping Lighthouse 1 normally.
2. Node A stops pinging Lighthouse 1.
3. Node A starts pinging Node B, but cannot ping it at this time.
4. Node A stops pinging Node B.
5. Node A pings Lighthouse 1 again, but cannot ping it at this time.

[QZAiXH]

My other two machines are experiencing the same issue.

One is a Windows system with a local network IP of 192.168.8.15, and the other is a Linux system with a local network IP of 192.168.8.34.

The Nebula IP for Windows is 172.17.168.189, and for Linux, it is 172.17.59.47.

Note that Windows uses a WiFi connection, while Linux uses a wired network. In any case, the Linux machine can ping both lighthouse nodes successfully.

The following operations were performed on the Windows machine:

1. Initially, pinging both lighthouse IPs works normally.
2. Disconnect the ping connection to the lighthouses.
3. Pinging the Linux node fails.
4. After the failure, pinging both lighthouse IPs becomes impossible.

The results of these operations are consistent whether on the local network or an external network environment.

windows configuration:

relay:
    relays:
        - 172.17.215.97
        - 172.17.0.1
    am_relay: false
    use_relays: true
tun:
    unsafe_routes: []
    disabled: false
    dev: fsnb0
    drop_local_broadcast: false
    drop_multicast: false
    tx_queue: 500
    mtu: 1300
    routes: []
logging:
    level: info
    format: text
firewall:
    outbound_action: reject
    inbound_action: reject
    conntrack:
        tcp_timeout: 12m
        udp_timeout: 3m
        default_timeout: 10m
    outbound:
        - port: any
          proto: any
          host: any
    inbound:
        - port: any
          proto: any
          host: 1EzzQ2DAEL
          groups: []
        - port: any
          proto: any
          host: pL0bCwBFbr
          groups: []
        - port: any
          proto: any
          host: 9BGqXzvo2W
          groups: []
        - port: any
          proto: any
          host: CvxMA7AmCb
          groups: []
        - port: any
          proto: any
          host: RpTxhV527F
          groups: []
        - port: any
          proto: any
          host: bvfbKA1wVl
          groups: []
        - port: any
          proto: any
          host: rBQbR4DZ8M
          groups: []
        - port: any
          proto: any
          host: fpy953QwVb
          groups: []
        - port: any
          proto: any
          host: Xo4EklX2R7
          groups: []
        - port: any
          proto: any
          host: a8Uw2D9t85
          groups: []
        - port: any
          proto: any
          host: Z75aqFT0xv
          groups: []
        - port: any
          proto: any
          host: tv0SNu0ypt
          groups: []
        - port: any
          proto: any
          host: R2X5xSZySO
          groups: []
        - port: any
          proto: any
          host: YJLlrfe7TY
          groups: []
        - port: any
          proto: any
          host: ufFp1RwNK6
          groups: []
        - port: any
          proto: any
          host: hVODrWRcJE
          groups: []
        - port: any
          proto: any
          host: 9capoEcIUB
          groups: []
        - port: any
          proto: any
          host: cyFwCAWLl4
          groups: []
        - port: any
          proto: any
          host: pJOEJeAEJg
          groups: []
        - port: any
          proto: any
          host: UpQEAWd2mf
          groups: []
        - port: any
          proto: any
          host: tXrNin8qfa
          groups: []
        - port: any
          proto: any
          host: SczMyvPc8U
          groups: []
        - port: any
          proto: any
          host: wQ3oHGfUpa
          groups: []
        - port: any
          proto: any
          host: mJZf55lGfu
          groups: []
        - port: any
          proto: any
          host: 3usVITf94h
          groups: []
        - port: any
          proto: any
          host: D2O7zlfF8f
          groups: []
        - port: any
          proto: any
          host: qqdNWqvLBt
          groups: []
        - port: any
          proto: any
          host: jdOsD0vXMR
          groups: []
        - port: any
          proto: icmp
          host: any
          groups: []
pki:
    ca: xxx
    cert: xxx
    key: xxx
    disconnect_invalid: true
static_host_map:
    172.17.0.1:
        - x.x.x.x:4242
    172.17.215.97:
        - xx.xx.xx.xx:4242
lighthouse:
    am_lighthouse: false
    hosts:
        - 172.17.0.1
        - 172.17.215.97
    interval: 60
    local_allow_list:
        interfaces: {}
    advertise_addrs: []
listen:
    host: 0.0.0.0
    port: 0
punchy:
    delay: 1s
    respond_delay: 5s
    punch: true
    respond: true
handshakes:
    trigger_buffer: 64
    try_interval: 100ms
    retries: 10
static_map:
    network: ip4
    cadence: 30s
    lookup_timeout: 250ms
cipher: aes
routines: 1

linux configuration:

relay:
    relays:
        - 172.17.215.97
        - 172.17.0.1
    am_relay: false
    use_relays: true
tun:
    unsafe_routes: []
    disabled: false
    dev: fsnb0
    drop_local_broadcast: false
    drop_multicast: false
    tx_queue: 500
    mtu: 1300
    routes: []
logging:
    level: info
    format: text
firewall:
    outbound_action: reject
    inbound_action: reject
    conntrack:
        tcp_timeout: 12m
        udp_timeout: 3m
        default_timeout: 10m
    outbound:
        - port: any
          proto: any
          host: any
    inbound:
        - port: any
          proto: any
          host: 1EzzQ2DAEL
          groups: []
        - port: any
          proto: any
          host: pL0bCwBFbr
          groups: []
        - port: any
          proto: any
          host: 9BGqXzvo2W
          groups: []
        - port: any
          proto: any
          host: CvxMA7AmCb
          groups: []
        - port: any
          proto: any
          host: RpTxhV527F
          groups: []
        - port: any
          proto: any
          host: bvfbKA1wVl
          groups: []
        - port: any
          proto: any
          host: rBQbR4DZ8M
          groups: []
        - port: any
          proto: any
          host: fpy953QwVb
          groups: []
        - port: any
          proto: any
          host: Xo4EklX2R7
          groups: []
        - port: any
          proto: any
          host: a8Uw2D9t85
          groups: []
        - port: any
          proto: any
          host: Z75aqFT0xv
          groups: []
        - port: any
          proto: any
          host: tv0SNu0ypt
          groups: []
        - port: any
          proto: any
          host: R2X5xSZySO
          groups: []
        - port: any
          proto: any
          host: YJLlrfe7TY
          groups: []
        - port: any
          proto: any
          host: ufFp1RwNK6
          groups: []
        - port: any
          proto: any
          host: hVODrWRcJE
          groups: []
        - port: any
          proto: any
          host: 9capoEcIUB
          groups: []
        - port: any
          proto: any
          host: cyFwCAWLl4
          groups: []
        - port: any
          proto: any
          host: pJOEJeAEJg
          groups: []
        - port: any
          proto: any
          host: UpQEAWd2mf
          groups: []
        - port: any
          proto: any
          host: tXrNin8qfa
          groups: []
        - port: any
          proto: any
          host: SczMyvPc8U
          groups: []
        - port: any
          proto: any
          host: wQ3oHGfUpa
          groups: []
        - port: any
          proto: any
          host: mJZf55lGfu
          groups: []
        - port: any
          proto: any
          host: 3usVITf94h
          groups: []
        - port: any
          proto: any
          host: D2O7zlfF8f
          groups: []
        - port: any
          proto: any
          host: qqdNWqvLBt
          groups: []
        - port: any
          proto: icmp
          host: any
          groups: []
pki:
    ca: xxx
    cert: xxx
    key: xxx
    disconnect_invalid: true
static_host_map:
    172.17.0.1:
        - x.x.x.x:4242
    172.17.215.97:
        - xx.xx.xx.xx:4242
lighthouse:
    am_lighthouse: false
    hosts:
        - 172.17.0.1
        - 172.17.215.97
    interval: 60
    local_allow_list:
        interfaces: {}
    advertise_addrs: []
listen:
    host: 0.0.0.0
    port: 0
punchy:
    delay: 1s
    respond_delay: 5s
    punch: true
    respond: true
handshakes:
    trigger_buffer: 64
    try_interval: 100ms
    retries: 10
static_map:
    network: ip4
    cadence: 30s
    lookup_timeout: 250ms
cipher: aes
routines: 1

l1-lan.log
l1-remote.log
l2-lan.log
l2-remote.log
node-linux-lan.log
node-linux-remote.log
node-windows-lan.log
node-windows-remote.log

[QZAiXH]

@johnmaguire How can I troubleshoot or resolve this situation?

[QZAiXH]

@johnmaguire Devices using WiFi have a higher probability of encountering this issue, which can be resolved by switching to a wired connection. Computers that rely on WiFi for extended periods are particularly prone to this situation.