Unhealthy Health check exceeded timeout #851

guiweber · 2024-11-04T16:16:09Z

The container works but reports as unhealthy due to the health check timing out.

Is this urgent: No
DNS provider(s) you use: Dreamhost
Program version: 2.8.1 built on 2024-09-20T07:18:44.173Z (commit c16287e)
What are you using to run the container: TrueNAS 24.10
Extra information:

Docker inspection reports Health check exceeded timeout (5s)

For reference, an issue was open on TrueNAS's github (truenas/apps#751), but they believe the issue is with the base DDNS-updated container.

Logs:

2024-11-04 15:29:26.843282+00:00========================================
2024-11-04 15:29:26.843309+00:00========================================
2024-11-04 15:29:26.843311+00:00============= ddns-updater =============
2024-11-04 15:29:26.843313+00:00========================================
2024-11-04 15:29:26.843315+00:00=========== Made with ❤️ by ============
2024-11-04 15:29:26.843320+00:00======= https://github.com/qdm12 =======
2024-11-04 15:29:26.843321+00:00========================================
2024-11-04 15:29:26.843323+00:00========================================
2024-11-04 15:29:26.843324+00:002024-11-04T15:29:26.843324944Z
2024-11-04 15:29:26.843326+00:00Running version 2.8.1 built on 2024-09-20T07:18:44.173Z (commit c16287e)
2024-11-04 15:29:26.843329+00:002024-11-04T15:29:26.843329252Z
2024-11-04 15:29:26.843330+00:00🔧 Need help? ☕ Discussion? https://github.com/qdm12/ddns-updater/discussions/new/choose
2024-11-04 15:29:26.843332+00:00🐛 Bug? ✨ New feature? https://github.com/qdm12/ddns-updater/issues/new/choose
2024-11-04 15:29:26.843334+00:00💻 Email? [email protected]
2024-11-04 15:29:26.843336+00:00💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2024-11-04 15:29:26.843403+00:002024-11-04T16:29:26+01:00 INFO Settings summary:
2024-11-04 15:29:26.843411+00:00├── HTTP client
2024-11-04 15:29:26.843414+00:00|   └── Timeout: 10s
2024-11-04 15:29:26.843419+00:00├── Update
2024-11-04 15:29:26.843421+00:00|   ├── Period: 5m0s
2024-11-04 15:29:26.843422+00:00|   └── Cooldown: 1m0s
2024-11-04 15:29:26.843424+00:00├── Public IP fetching
2024-11-04 15:29:26.843425+00:00|   ├── HTTP enabled: yes
2024-11-04 15:29:26.843427+00:00|   ├── HTTP IP providers
2024-11-04 15:29:26.843430+00:00|   |   └── all
2024-11-04 15:29:26.843432+00:00|   ├── HTTP IPv4 providers
2024-11-04 15:29:26.843434+00:00|   |   └── all
2024-11-04 15:29:26.843435+00:00|   ├── HTTP IPv6 providers
2024-11-04 15:29:26.843437+00:00|   |   └── all
2024-11-04 15:29:26.843440+00:00|   ├── DNS enabled: yes
2024-11-04 15:29:26.843442+00:00|   ├── DNS timeout: 3s
2024-11-04 15:29:26.843443+00:00|   └── DNS over TLS providers
2024-11-04 15:29:26.843445+00:00|       └── all
2024-11-04 15:29:26.843447+00:00├── Resolver: use Go default resolver
2024-11-04 15:29:26.843449+00:00├── Server
2024-11-04 15:29:26.843451+00:00|   ├── Listening address: :30007
2024-11-04 15:29:26.843453+00:00|   └── Root URL: /
2024-11-04 15:29:26.843454+00:00├── Health
2024-11-04 15:29:26.843456+00:00|   └── Server listening address: 0.0.0.0:9999
2024-11-04 15:29:26.843458+00:00├── Paths
2024-11-04 15:29:26.843460+00:00|   ├── Data directory: /updater/data
2024-11-04 15:29:26.843462+00:00|   ├── Config file: /updater/data/config.json
2024-11-04 15:29:26.843463+00:00|   └── Umask: system default
2024-11-04 15:29:26.843465+00:00├── Backup
2024-11-04 15:29:26.843467+00:00|   ├── Period: 24h0m0s
2024-11-04 15:29:26.843469+00:00|   └── Directory: /updater/data
2024-11-04 15:29:26.843470+00:00└── Logger
2024-11-04 15:29:26.843472+00:00├── Level: info
2024-11-04 15:29:26.843473+00:00└── Caller: hidden
2024-11-04 15:29:26.843477+00:002024-11-04T16:29:26+01:00 INFO reading JSON config from environment variable CONFIG
2024-11-04 15:29:26.843553+00:002024-11-04T16:29:26+01:00 INFO Found 5 settings to update records
2024-11-04 15:29:34.907146+00:002024-11-04T16:29:34+01:00 INFO Reading history from database: domain stematics.net owner media ipv4
2024-11-04 15:29:34.907186+00:002024-11-04T16:29:34+01:00 INFO Reading history from database: domain stematics.net owner drive ipv4
2024-11-04 15:29:34.907201+00:002024-11-04T16:29:34+01:00 INFO Reading history from database: domain stematics.net owner play ipv4
2024-11-04 15:29:34.907204+00:002024-11-04T16:29:34+01:00 INFO Reading history from database: domain stematics.net owner homectl ipv4
2024-11-04 15:29:34.907206+00:002024-11-04T16:29:34+01:00 INFO Reading history from database: domain stematics.net owner home ipv4
2024-11-04 15:29:34.907532+00:002024-11-04T16:29:34+01:00 INFO [healthcheck server] health http server listening on [::]:9999
2024-11-04 15:29:34.907557+00:002024-11-04T16:29:34+01:00 INFO [http server] http server listening on [::]:30007
2024-11-04 15:29:34.907571+00:002024-11-04T16:29:34+01:00 INFO [backup] each 24h0m0s; writing zip files to directory /updater/data
2024-11-04 15:29:39.959768+00:002024-11-04T16:29:39+01:00 INFO obtaining ipv4 address succeeded after 1 failed try
2024-11-04 15:34:39.955882+00:002024-11-04T16:34:39+01:00 INFO obtaining ipv4 address succeeded after 1 failed try
2024-11-04 15:39:39.962088+00:002024-11-04T16:39:39+01:00 INFO obtaining ipv4 address succeeded after 1 failed try
2024-11-04 15:44:39.952701+00:002024-11-04T16:44:39+01:00 INFO obtaining ipv4 address succeeded after 1 failed try
2024-11-04 15:49:39.958142+00:002024-11-04T16:49:39+01:00 INFO obtaining ipv4 address succeeded after 1 failed try
2024-11-04 15:54:39.955310+00:002024-11-04T16:54:39+01:00 INFO obtaining ipv4 address succeeded after 1 failed try
2024-11-04 15:59:15.003662+00:002024/11/04 16:59:15 "GET http://192.168.1.17:30007/ HTTP/1.1" from 192.168.1.34:33232 - 200 4832B in 958.801µs
2024-11-04 15:59:15.034121+00:002024/11/04 16:59:15 "GET http://192.168.1.17:30007/static/styles.css HTTP/1.1" from 192.168.1.34:33232 - 200 3203B in 530.707µs
2024-11-04 15:59:15.081702+00:002024/11/04 16:59:15 "GET http://192.168.1.17:30007/static/favicon.svg HTTP/1.1" from 192.168.1.34:33232 - 200 3192B in 67.176µs
2024-11-04 15:59:39.961502+00:002024-11-04T16:59:39+01:00 INFO obtaining ipv4 address succeeded after 1 failed try
2024-11-04 16:04:40.020689+00:002024-11-04T17:04:40+01:00 INFO obtaining ipv4 address succeeded after 1 failed try

Docker inspection:

{
  "Status": "running",
  "Running": true,
  "Paused": false,
  "Restarting": false,
  "OOMKilled": false,
  "Dead": false,
  "Pid": 49123,
  "ExitCode": 0,
  "Error": "",
  "StartedAt": "2024-11-04T15:29:26.666736557Z",
  "FinishedAt": "0001-01-01T00:00:00Z",
  "Health": {
    "Status": "unhealthy",
    "FailingStreak": 127,
    "Log": [
      {
        "Start": "2024-11-04T17:00:12.243452166+01:00",
        "End": "2024-11-04T17:00:17.282567472+01:00",
        "ExitCode": -1,
        "Output": "Health check exceeded timeout (5s)"
      },
      {
        "Start": "2024-11-04T17:00:27.283142218+01:00",
        "End": "2024-11-04T17:00:32.326351763+01:00",
        "ExitCode": -1,
        "Output": "Health check exceeded timeout (5s)"
      },
      {
        "Start": "2024-11-04T17:00:42.327172058+01:00",
        "End": "2024-11-04T17:00:47.382602929+01:00",
        "ExitCode": -1,
        "Output": "Health check exceeded timeout (5s)"
      },
      {
        "Start": "2024-11-04T17:00:57.383121142+01:00",
        "End": "2024-11-04T17:01:02.423358376+01:00",
        "ExitCode": -1,
        "Output": "Health check exceeded timeout (5s)"
      },
      {
        "Start": "2024-11-04T17:01:12.424394643+01:00",
        "End": "2024-11-04T17:01:17.4709455+01:00",
        "ExitCode": -1,
        "Output": "Health check exceeded timeout (5s)"
      }
    ]
  }
}

Configuration file (remove your credentials!):

{
  "settings": [
    {
      "domain": "media.stematics.net",
      "ip_version": "ipv4",
      "key": "",
      "provider": "dreamhost"
    },
    {
      "domain": "drive.stematics.net",
      "ip_version": "ipv4",
      "key": "",
      "provider": "dreamhost"
    },
    {
      "domain": "play.stematics.net",
      "ip_version": "ipv4",
      "key": "",
      "provider": "dreamhost"
    },
    {
      "domain": "homectl.stematics.net",
      "ip_version": "ipv4",
      "key": "",
      "provider": "dreamhost"
    },
    {
      "domain": "home.stematics.net",
      "ip_version": "ipv4",
      "key": "",
      "provider": "dreamhost"
    }
  ]
}

The text was updated successfully, but these errors were encountered:

marianozunino · 2024-11-05T14:22:17Z

I disabled the health check for now; otherwise, Traefik wouldn’t forward requests to the service.

services:
  ddns:
    restart: unless-stopped
    image: qmcgaw/ddns-updater
    container_name: ddns
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
    volumes:
      - ./config:/config
      - ./data:/updater/data
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.ddns.rule=Host(`ddns.${LAN_DOMAIN}`)"
      - "traefik.http.routers.ddns.entrypoints=web"
      - "traefik.http.services.ddns.loadbalancer.server.port=8000"
    healthcheck:
      disable: true

gresnyer · 2024-11-06T13:00:27Z

I have the same issue with TrueNas Scale ElectricEel-24.10.0.
When I deploying this from Apps it remains in "Deploying" state forever. In this case I don't have an option to disable health check. However if I adding as a custom app I can disable health check and reaching "Running" state.

I recommend to fix health check functionality.
If it will be fixed I will switch back and will use from Apps directly.

qdm12 · 2024-11-19T22:53:14Z

The healthcheck is working fine on my side, and reading the debugging steps you went through, it looks like everything I would had tried. I'm suspecting a problem with the host firewall or Docker runtime 🤔

I spent my day re-working/updating https://github.com/qdm12/port-checker such that you can now specify a listening address instead of just a listening port; can you try then:

sudo docker run -it --rm --name test alpine:3.20
wget -qO port-checker https://github.com/qdm12/port-checker/releases/download/v0.4.0/port-checker_0.4.0_linux_amd64
chmod +x port-checker
./port-checker --listening-address="127.0.0.1:9999"

(in case your CPU architecture isn't amd64, you can find the right one at https://github.com/qdm12/port-checker/releases/tag/v0.4.0)

And in another terminal, at the same time:

sudo docker exec test wget -qO- http://127.0.0.1:9999

and check if this works?

You can then exit the port-checker program by hitting CTRL+C and then exit the test container with exit.

qdm12 added Status: 🗯️ Waiting for feedback Category: Healthcheck 🩺 labels Nov 21, 2024

qdm12 changed the title ~~Health check timeout~~ Unhealthy Health check exceeded timeout Nov 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unhealthy Health check exceeded timeout #851

Unhealthy Health check exceeded timeout #851

guiweber commented Nov 4, 2024

marianozunino commented Nov 5, 2024

gresnyer commented Nov 6, 2024

qdm12 commented Nov 19, 2024

Unhealthy Health check exceeded timeout #851

Unhealthy Health check exceeded timeout #851

Comments

guiweber commented Nov 4, 2024

marianozunino commented Nov 5, 2024

gresnyer commented Nov 6, 2024

qdm12 commented Nov 19, 2024