Dozzle in swarm not connecting to agent

[michaelmegrath]

🔍 Check for existing issues

• Completed

How is Dozzle deployed?

Swarm Mode

📦 Dozzle version

8.10.0

✅ Command used to run Dozzle

Swarm config:

services:
  dozzle:
    image: amir20/dozzle:latest
    environment:
      - DOZZLE_MODE=swarm
      - DOZZLE_REMOTE_AGENT=10.0.0.220:7007
      - DOZZLE_LEVEL=debug
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - 8080:8080
    networks:
      - swarm-net
    deploy:
      mode: global
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.dozzle.entrypoints=websecure"
        - "traefik.http.routers.dozzle.rule=Host(`dozzle.domain.com`) || Host(`dozzle.domain2.com`)"
        - "traefik.http.routers.dozzle.tls=true" 
        - "traefik.http.services.dozzle.loadbalancer.server.port=8080"

networks:
  swarm-net:
    external: true

Agent Config:

services:
  dozzle-agent:
    image: amir20/dozzle:latest
    command: agent
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - 7007:7007

🐛 Describe the bug / provide steps to reproduce it

Sorry if this is too similar to Issue #3344. There was no resolution for that issue and I am running in swarm mode which may be part of the problem potentially.

All my container in my docker swarm are configured and working properly. I am trying to connect a host not on the docker swarm via the agent. It says the host is offline.

I have from a container in the same network/swarm as the dozzle server run this:

fa9c77bb6918:/config# nc -zv 10.0.0.220 7007
10.0.0.220 (10.0.0.220:7007) open

So it doesn't appear to be a firewall issue, which has me stumped.

💻 Environment

Dozzle Server's Environment

Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 15
  Running: 12
  Paused: 0
  Stopped: 3
 Images: 15
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: b81iil83oghyt5cbwzjhvgs7p
  Is Manager: true
  ClusterID: 0op70i0f2m59i8qhd7unl50ng
  Managers: 2
  Nodes: 4
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.0.0.228
  Manager Addresses:
   10.0.0.228:2377
   10.0.0.238:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-1013-raspi
 Operating System: Ubuntu 24.04.1 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.752GiB
 Name: host1
 ID: 08822b6f-63ac-4052-95e6-379f56fc5430
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: Running Swarm in a two-manager configuration. This configuration provides
         no fault tolerance, and poses a high risk to lose control over the cluster.
         Refer to https://docs.docker.com/engine/swarm/admin_guide/ to configure the
         Swarm for fault-tolerance.

Dozzle agent's Environment

sudo docker system info
Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 11
  Running: 9
  Paused: 0
  Stopped: 2
 Images: 35
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.11.0-1004-raspi
 Operating System: Ubuntu 24.10
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.751GiB
 Name: host2
 ID: cf7d5348-d8e1-4b40-ae2e-bf0b0c6e45c2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

📸 If applicable, add screenshots to help explain your bug

📜 If applicable, attach your Dozzle logs. You many need to enable debug mode. See https://dozzle.dev/guide/debugging.

{
"level": "warn",
"error": "rpc error: code = DeadlineExceeded desc = received context error while waiting for new LB policy update: context deadline exceeded",
"endpoint": "10.0.0.220:7007",
"time": "2025-01-07T00:01:10Z",
"message": "error fetching host info for agent"
}

[amir20]

It all looks right here. There is a testing subcommand you can use:

$ docker run amir20/dozzle:latest agent-test 10.0.0.220:7007

Run that to see what the error is. If it succeeds, then something is wrong with the swarm network. If it doesn't work, then the agent is not set up correctly or is behind a firewall.

[michaelmegrath]

Command was run on the same host as dozzle, but not within the docker network that dozzle is in:

$ docker run amir20/dozzle:latest agent-test 10.0.0.220:7007
Unable to find image 'amir20/dozzle:latest' locally
latest: Pulling from amir20/dozzle
Digest: sha256:ef2f2e3727a1ee01b9d6cfd4d53b1589b2b8a9b8405775f5027f0481a2c19201
Status: Downloaded newer image for amir20/dozzle:latest
{"level":"info","endpoint":"10.0.0.220:7007","time":"2025-01-07T05:06:51Z","message":"Connecting to agent"}
{"level":"info","endpoint":"10.0.0.220:7007","version":"v8.10.0","name":"host2","id":"cf7d5348-d8e1-4b40-ae2e-bf0b0c6e45c2","time":"2025-01-07T05:06:51Z","message":"Successfully connected to agent"}

Here is my attempt to run the command on the docker swarm network I have

$ docker run --network=swarm-net amir20/dozzle:latest agent-test 10.0.0.220:7007
docker: Error response from daemon: Could not attach to network swarm-net: rpc error: code = PermissionDenied desc = network swarm-net not manually attachable.

Here is more information about said network if it is helpful at all:

[
    {
        "Name": "swarm-net",
        "Id": "yfqx1qstsajdqy4btsi77gq5q",
        "Created": "2024-11-06T17:01:49.356851387Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "2595a0665995ffa3f0b36e534f16b6bf84e5de0457776a524eb1305ddbbfe4cb": {
                "Name": "immich_immich-server.1.myq1syvejiji955gcpe0ekzpm",
                "EndpointID": "68845fa36a1600c7e8a209e5d51b5b9957f14775d0939306fe60b0d72c26d582",
                "MacAddress": "02:42:0a:00:01:77",
                "IPv4Address": "10.0.1.119/24",
                "IPv6Address": ""
            },
            "28bdfd82c63670c212e5e8e476060f6fa386a726f69dfb2cbad5e4e1c264d17d": {
                "Name": "dozzle_dozzle.b81iil83oghyt5cbwzjhvgs7p.jskqp0sv1l6sgr3spfww8jwyd",
                "EndpointID": "3ab7fee11608a28d755fb7e63dabed0341554960728540650a0a18698e4d1f08",
                "MacAddress": "02:42:0a:00:01:b0",
                "IPv4Address": "10.0.1.176/24",
                "IPv6Address": ""
            },
            "4426dd1f1c41fda9feb7386e325ce70cf3163ec99efcb41e604ccd0fba5af225": {
                "Name": "immich_redis.1.smm7bkypj8ngokxnv3588z091",
                "EndpointID": "04d9084e5d5d40fbb082adf55e363b49d91eac96d4c7519a3190998fe048be6c",
                "MacAddress": "02:42:0a:00:01:15",
                "IPv4Address": "10.0.1.21/24",
                "IPv6Address": ""
            },
            "573ad656d3e1a5b2cf4f9e7146ce4d1ba7e0c4118f7e4da080cd09690555aed9": {
                "Name": "immich_database.1.3blju8vdy9mmdpdg9r8aco6df",
                "EndpointID": "c4c3c373e2363687cf2ba0a4d4a66429974575cafb63119dfb17a35c035b8315",
                "MacAddress": "02:42:0a:00:01:18",
                "IPv4Address": "10.0.1.24/24",
                "IPv6Address": ""
            },
            "61fe3e3af3f41d390de7601ac13c4e426b53511ef4d1f3a4d7c9f88403694394": {
                "Name": "traefik_traefik.b81iil83oghyt5cbwzjhvgs7p.r298o5on87s0ztmr1xrevloyy",
                "EndpointID": "ff76bcc1fc5a74bfca4697ffbfb7c66cf4198c19e844538bb8447fd9fb0a9d25",
                "MacAddress": "02:42:0a:00:01:28",
                "IPv4Address": "10.0.1.40/24",
                "IPv6Address": ""
            },
            "6308bd2f2c67318ea85015c0783a69c638c14cbd55bc65013706396805855d25": {
                "Name": "homepage_local.1.5nok2jsfwyzalsx12um4hgzxo",
                "EndpointID": "9e1f68380b9071b3cab99b4976f8a0f8796ff9c548949e56a3a4ebde7c8af8bb",
                "MacAddress": "02:42:0a:00:01:53",
                "IPv4Address": "10.0.1.83/24",
                "IPv6Address": ""
            },
            "93ade4d789915571c3ca6a95399c3f84a0b01dd9c697942addf1664202efe1c3": {
                "Name": "homepage_remote.1.pyul8u1jf4lz4uh4lp6it8jx0",
                "EndpointID": "79d9bfc5e23f90df5f7eda26b7d69c2ec9a630482a9afd31ae3e5cc8dc3cbf98",
                "MacAddress": "02:42:0a:00:01:55",
                "IPv4Address": "10.0.1.85/24",
                "IPv6Address": ""
            },
            "c9f96ca10fc2d5764508668aa128c09998c45595af3faa444c59cd3b0264bee5": {
                "Name": "portainer_portainer.1.vfbrtx6xu55gw1kubjv7l4kao",
                "EndpointID": "15886cea20c550c32dac6541bf46c54b02d55a9f48f0f7ff08c34230ddbda74a",
                "MacAddress": "02:42:0a:00:01:45",
                "IPv4Address": "10.0.1.69/24",
                "IPv6Address": ""
            },
            "fa9c77bb6918601352e7c5afba9f2de47e8413d37eaba3af374824a02aa7c886": {
                "Name": "homeassistant_homeassistant.1.gam2lio33ilswk5b0qo0in618",
                "EndpointID": "ac1f7adcb06d4978bd970da18fa30570ba020df9439c867426173c03cacb7aaa",
                "MacAddress": "02:42:0a:00:01:48",
                "IPv4Address": "10.0.1.72/24",
                "IPv6Address": ""
            },
            "fcfea9de5cf97cda27fc1b95ff5f4c4f7510e0bbc719da9854b84e5e8f0af647": {
                "Name": "immich_immich-machine-learning.1.pi3mlhs2ci4dch59ifz1wadle",
                "EndpointID": "76eae95b6a505212b9c6c188a9e293025f70a61d9034e72a23d64ee1c372c079",
                "MacAddress": "02:42:0a:00:01:1e",
                "IPv4Address": "10.0.1.30/24",
                "IPv6Address": ""
            },
            "lb-swarm-net": {
                "Name": "swarm-net-endpoint",
                "EndpointID": "f5e3c8da9d863d50f72711c15ccb74a32b87d50e14eb6e85ec4166181e1b0700",
                "MacAddress": "02:42:0a:00:01:1d",
                "IPv4Address": "10.0.1.29/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "078a94419b02",
                "IP": "10.0.0.228"
            },
            {
                "Name": "60228e2b4016",
                "IP": "10.0.0.238"
            },
            {
                "Name": "e3d340bf8cc4",
                "IP": "10.0.0.78"
            },
            {
                "Name": "4183f2c9e12b",
                "IP": "10.0.0.218"
            }
        ]
    }
]

[amir20]

Seems like without network it works.

docker: Error response from daemon: Could not attach to network swarm-net: rpc error: code = PermissionDenied desc = network swarm-net not manually attachable.

This is actually a docker error. You can't attach to networks unless attachable is enabled. I imagine if the above command worked then this worked too.

How are you creating the network? Did you enable --internal?

I can try to bring up my test swarm network this week and see if it makes a difference. But I am pretty sure it has something to do with your network. 😕

[michaelmegrath]

I fixed the issue! At first I updated the docker network to set the "attachable" flag to true, and ran the testing command again to find that it was able to connect:

sudo docker run --network=swarm-net amir20/dozzle:latest agent-test 10.0.0.220:7007
{"level":"info","endpoint":"10.0.0.220:7007","time":"2025-01-07T16:23:50Z","message":"Connecting to agent"}
{"level":"info","endpoint":"10.0.0.220:7007","version":"v8.10.0","name":"host2","id":"cf7d5348-d8e1-4b40-ae2e-bf0b0c6e45c2","time":"2025-01-07T16:23:50Z","message":"Successfully connected to agent"}

I then decided to take a closer look at the docker-compose file I had setup and I began removing unnecessary config lines to simplify the troubleshooting. The first lines I removed was

    ports:
      - 8080:8080

I realized I didn't actual need to expose the web-gui port since traefik acts as my reverse proxy hence why I removed these lines. But when I brought up the container it appears to have fixed the connection to the agent. It is unclear why exposing the web-gui port prevented the server from connecting to the agent though.

[amir20]

I doubt that made a difference. Maybe something else happened. But good to hear things are good now.