Windows host is unreachable from inside container or podman machine

[maxirass]

Issue Description

I am trying to reach my host from a container (which I use for openapi-generator).
I have already tried different flags and settings.
Neither host.containers.internal nor --network=host can I access the swagger api on localhost:5001.
If I execute ps auxww | grep pasta in the podman-machine-default:

user         626  0.0  0.0  68476 24308 ?        Ss   10:48   0:00 /usr/bin/pasta --config-net --dns-forward 169.254.1.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/1000/netns/netns-ef11a5a8-056d-9e22-cfdc-83fa701bceae --map-guest-addr 169.254.1.2
user         734  0.0  0.0   3952  1880 pts/1    S+   10:56   0:00 grep --color=auto pasta

I start an alpine container:
podman run --rm -it alpine sh

Then these are the following outputs:

These is the ip route output inside the podman machine:

ip route
default via 172.29.32.1 dev eth0 proto kernel
172.29.32.0/20 dev eth0 proto kernel scope link src 172.29.37.89

I have the problem with Podman v 5.0.0 / 5.3.0 and 5.3.x

The only thing that has worked so far is using
networkingMode=mirrored in the .wslconfig
and starting the container with --network=host.
Then I can access the swagger api directly with localhost or 127.0.0.1. However, this causes other problems with my DB, which is why this is not an option!

Steps to reproduce the issue

1. Create default podman machine
2. Create standard alpine container and install curl
3. Some service, like swagger api, is running on the host, e.g. localhost:5001 or 127.0.0.1:5001
4. Try to curl the swagger.json file

Describe the results you received

From inside the container!

 curl -v https://host.containers.internal:5001/swagger/v1/swagger.json
* Host host.containers.internal:5001 was resolved.
* IPv6: (none)
* IPv4: 169.254.1.2
*   Trying 169.254.1.2:5001...
* connect to 169.254.1.2 port 5001 from 172.29.37.89 port 42238 failed: Operation timed out
* Failed to connect to host.containers.internal port 5001 after 131710 ms: Could not connect to server
* closing connection #0
curl: (28) Failed to connect to host.containers.internal port 5001 after 131710 ms: Could not connect to server

or

nslookup host.containers.internal
Server:         169.254.1.1
Address:        169.254.1.1:53

** server can't find host.containers.internal: SERVFAIL

** server can't find host.containers.internal: SERVFAIL

but when I ping:

ping host.containers.internal:5001
PING host.containers.internal:5001 (169.254.1.2): 56 data bytes
64 bytes from 169.254.1.2: seq=0 ttl=42 time=0.974 ms
64 bytes from 169.254.1.2: seq=1 ttl=42 time=0.445 ms
64 bytes from 169.254.1.2: seq=2 ttl=42 time=0.449 ms

Describe the results you expected

Host service, like a swagger api, should be reachable from inside the container via
host.containers.internal

Output should be similar to the curl command from the windows host (powershell):

 curl -v https://localhost:5001/swagger/v1/swagger.json
* Host localhost:5001 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:5001...
* Connected to localhost (::1) port 5001
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.x
> GET /swagger/v1/swagger.json HTTP/1.1
> Host: localhost:5001
> User-Agent: curl/8.9.1
> Accept: */*

podman info output

host:
  arch: amd64
  buildahVersion: 1.38.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.12-2.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 99.62
    systemPercent: 0.25
    userPercent: 0.13
  cpus: 12
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "40"
  eventLogger: journald
  freeLocks: 2048
  hostname: PASO-070
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 5.15.167.4-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 41296322560
  memTotal: 42063781888
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.13.1-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.1
    package: netavark-1.13.1-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.1
  ociRuntime:
    name: crun
    package: crun-1.19.1-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.19.1
      commit: 3e32a70c93f5aa5fea69b50256cca7fd4aa23c80
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20241211.g09478d5-1.fc40.x86_64
    version: |
      pasta 0^20241211.g09478d5-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: unix:///run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 32212254720
  swapTotal: 32212254720
  uptime: 0h 7m 16.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 942125056
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 5.3.1
  Built: 1732147200
  BuiltTime: Thu Nov 21 01:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/amd64
  Version: 5.3.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

wsl --version
WSL-Version: 2.3.26.0
Kernelversion: 5.15.167.4-1
WSLg-Version: 1.0.65
MSRDC-Version: 1.2.5620
Direct3D-Version: 1.611.1-81528511
DXCore-Version: 10.0.26100.1-240331-1435.ge-release
Windows-Version: 10.0.22631.4751

No networkingMode changed

Additional information

No response

[maxirass]

@Luap99 today I tested the 5.4.0-rc3. If I step into the podman-machine with wsl -d then I can access the swagger api running on my localhost of the windows machine via the host.containers.internal

This is inside the podman-machine default:

curl -vk https://host.containers.internal:5001/swagger/v1/swagger.json
* Host host.containers.internal:5001 was resolved.
* IPv6: (none)
* IPv4: 192.168.127.254
*   Trying 192.168.127.254:5001...
* Connected to host.containers.internal (192.168.127.254) port 5001
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=localhost
*  start date: May  8 10:01:55 2024 GMT
*  expire date: May  8 10:01:55 2025 GMT
*  issuer: CN=localhost
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://host.containers.internal:5001/swagger/v1/swagger.json
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: host.containers.internal:5001]
* [HTTP/2] [1] [:path: /swagger/v1/swagger.json]
* [HTTP/2] [1] [user-agent: curl/8.6.0]
* [HTTP/2] [1] [accept: */*]
> GET /swagger/v1/swagger.json HTTP/2
> Host: host.containers.internal:5001
> User-Agent: curl/8.6.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200
< content-type: application/json;charset=utf-8
< date: Mon, 10 Feb 2025 09:31:09 GMT
< server: Kestrel
< strict-transport-security: max-age=2592000

But if i start a test alpine container and try the same thing i get this:

podman run --rm -it alpine /bin/sh

/ # curl -vk https://host.containers.internal:5001/swagger/v1/swagger.json
* Host host.containers.internal:5001 was resolved.
* IPv6: (none)
* IPv4: 169.254.1.2
*   Trying 169.254.1.2:5001...
* connect to 169.254.1.2 port 5001 from 192.168.127.2 port 36412 failed: Operation timed out
* Failed to connect to host.containers.internal port 5001 after 132400 ms: Could not connect to server
* closing connection #0
curl: (28) Failed to connect to host.containers.internal port 5001 after 132400 ms: Could not connect to server

Any further ideas?

[maxirass]

@Luap99 I found a very strange behaviour with the alpine test container.
If I use host.containers.internal it doesn't work properly. Then I checked which IP is referenced with host.containers.internal. If I use this IP in the curl command, every thing works correctly!

 podman run --rm -it alpine /bin/sh

 #  curl -vk https://host.containers.internal:5001/swagger/index.html
* Host host.containers.internal:5001 was resolved.
* IPv6: (none)
* IPv4: 169.254.1.2
*   Trying 169.254.1.2:5001...
* connect to 169.254.1.2 port 5001 from 192.168.127.2 port 60258 failed: Operation timed out
* Failed to connect to host.containers.internal port 5001 after 130699 ms: Could not connect to server
* closing connection #0
curl: (28) Failed to connect to host.containers.internal port 5001 after 130699 ms: Could not connect to server

/ # nslookup host.containers.internal
Server:         169.254.1.1
Address:        169.254.1.1:53

Non-authoritative answer:

Non-authoritative answer:
Name:   host.containers.internal
Address: 192.168.127.254

/ #  curl -vk https://192.168.127.254:5001/swagger/index.html
*   Trying 192.168.127.254:5001...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=localhost
*  start date: May  8 10:01:55 2024 GMT
*  expire date: May  8 10:01:55 2025 GMT
*  issuer: CN=localhost
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to 192.168.127.254 (192.168.127.254) port 5001
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://192.168.127.254:5001/swagger/index.html
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 192.168.127.254:5001]
* [HTTP/2] [1] [:path: /swagger/index.html]
* [HTTP/2] [1] [user-agent: curl/8.12.0]
* [HTTP/2] [1] [accept: */*]
> GET /swagger/index.html HTTP/2
> Host: 192.168.127.254:5001
> User-Agent: curl/8.12.0
> Accept: */*

[maxirass]

As an additional info. I had to activate the user-mode networking for the podman machine, since without this flag, the containers doesn't have any internet connection anymore with the new 5.4.0-rc3 version.

[Luap99]

169.254.1.2

That is the expected ip for the pasta network mode. The ip is mapped to the external ip in WSL.
If that is not further mapped to the windows host then we need to do something else on windows.

I don't use windows and have no idea about WSL networking (other that I know it is "broken", i.e. not behaving like a normal linux kernel) as such I cannot help you.

nslookup host.containers.internal

That does normally not work because we only add this to /etc/hosts. For podman machine on linux/macos and hyperV we use this via gvproxy to map to the actual host ip and not just the VM ip so the containers reach the real host. I assume user-mode networking runs the same gvproxy so it offers the same ip which seems to work.

If there is no networking at all since 5.4.0-rc3 without user mode networking then something seemsy broken, we do run CI tests on windows and they pass so I doubt it is a general problem. You could try building from source and use git bisect to fine the change on podman, if it actually is something in podman.
cc @l0rd

[maxirass]

connect to 169.254.1.2 port 5001 from 192.168.127.2 port 60258

I don't understand why the ip of the pasta network references not the same IP as the nslookup

Address: 192.168.127.25

and such a random port.
I neither unterstand where this 192.168.127.X is comming from.
This is my ipconfig from the windows host:

Drahtlos-LAN-Adapter Wi-Fi:

   Verbindungsspezifisches DNS-Suffix: localdomain
   Verbindungslokale IPv6-Adresse  . : fe80::d1ae:1baf:6c6a:2cf%28
   IPv4-Adresse  . . . . . . . . . . : 192.168.1.102
   Subnetzmaske  . . . . . . . . . . : 255.255.255.0
   Standardgateway . . . . . . . . . : 192.168.1.1

Ethernet-Adapter vEthernet (Default Switch):

   Verbindungsspezifisches DNS-Suffix:
   Verbindungslokale IPv6-Adresse  . : fe80::42e6:2ee1:4764:e4eb%6
   IPv4-Adresse  . . . . . . . . . . : 172.23.192.1
   Subnetzmaske  . . . . . . . . . . : 255.255.240.0
   Standardgateway . . . . . . . . . :

Ethernet-Adapter vEthernet (WSL (Hyper-V firewall)):

   Verbindungsspezifisches DNS-Suffix:
   Verbindungslokale IPv6-Adresse  . : fe80::13ca:6d33:d3fa:6d6d%60
   IPv4-Adresse  . . . . . . . . . . : 172.29.32.1
   Subnetzmaske  . . . . . . . . . . : 255.255.240.0
   Standardgateway . . . . . . . . . 

Hopefully I can describe the strange behaviour understandable.

[Luap99]

192.168.127.254

That is just an internal ip of gvproxy, like the (--map-guest-addr 169.254.1.2) ip of pasta it doesn't actually exists anywhere on an interface but is remapped to the host so it allows for connections from within. But from the outside these addresses are not reachable.

[maxirass]

Ok now I understand. So then the problem will be in the gvproxy during the NAT or proxy process?
Thats why curl command works not properly with host.containers.internal, because the proxy directs to the wrong IP or Port?
Am I correct? Or do you need further infos about this bug?

[maxirass]

@Luap99 @l0rd
I tested some other settings today. If I set the flag --add-host=host.containers.internal:192.168.127.254 and than I can use the host.containers.internal inside the container for curl.
If I do this via the --map-guest-addr,192.168.127.254 the host.containers.internal doesn't work.
But its interesting that in both ways the /etc/hosts file looks completely similar.
Do I use this in a wrong way or is it buggy with WSL2?

Also this whole add-host only works if the user-mode networking is activated!

I tested this with the stable realase 5.3.2

[l0rd]

I can confirm the problem. Inside a container running in a WSL VM, host.containers.internal is resolved to 169.254.1.2, and that's consistent with the entry in /etc/hosts/:

169.254.1.2     host.containers.internal host.docker.internal

And 169.254.1.2 doesn't get forwarded to the host IP. @Luap99 do you have an idea where are we testing this?