immich v1.120.2 keep restarting and high CPU usage

[henryxrl]

The bug

I just updated to v1.120.2, but immich_server keeps restarting and occupies 100% CPU.

The OS that Immich Server is running on

Ubuntu 22

Version of Immich Server

v1.120.2

Version of Immich Mobile App

v1.120.2

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2ba50e1ac3a0ea17b736ce9db2b0a9f6f8b85d4c27d5f5accc6a416d8f42c6d5
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command:
      [
        'postgres',
        '-c',
        'shared_preload_libraries=vectors.so',
        '-c',
        'search_path="$$user", public, vectors',
        '-c',
        'logging_collector=on',
        '-c',
        'max_wal_size=2GB',
        '-c',
        'shared_buffers=512MB',
        '-c',
        'wal_compression=on',
      ]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=America/New_York

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=xxxxx

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. docker rmi xxxxxxx (I first removed all the immich images)
2. docker compose up -d --pull always (always pull the latest version)
3. After a while, CPU reaches 100% usage.
   ...

Relevant log output

immich_postgres  |
immich_postgres  | PostgreSQL Database directory appears to contain a database; Skipping initialization
immich_postgres  |
immich_postgres  | 2024-11-13 17:01:48.852 UTC [1] LOG:  redirecting log output to logging collector process
immich_postgres  | 2024-11-13 17:01:48.852 UTC [1] HINT:  Future log output will appear in directory "log".
immich_machine_learning  | [11/13/24 12:01:50] INFO     Starting gunicorn 23.0.0
immich_machine_learning  | [11/13/24 12:01:50] INFO     Listening at: http://[::]:3003 (9)
immich_machine_learning  | [11/13/24 12:01:50] INFO     Using worker: app.config.CustomUvicornWorker
immich_redis             | 1:C 13 Nov 2024 17:01:48.615 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
immich_redis             | 1:C 13 Nov 2024 17:01:48.615 # Redis version=6.2.16, bits=64, commit=00000000, modified=0, pid=1, just started
immich_redis             | 1:C 13 Nov 2024 17:01:48.615 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
immich_redis             | 1:M 13 Nov 2024 17:01:48.615 * monotonic clock: POSIX clock_gettime
immich_redis             | 1:M 13 Nov 2024 17:01:48.617 * Running mode=standalone, port=6379.
immich_redis             | 1:M 13 Nov 2024 17:01:48.617 # Server initialized
immich_machine_learning  | [11/13/24 12:01:50] INFO     Booting worker with pid: 10
immich_machine_learning  | [11/13/24 12:02:02] INFO     Started server process [10]
immich_machine_learning  | [11/13/24 12:02:02] INFO     Waiting for application startup.
immich_machine_learning  | [11/13/24 12:02:02] INFO     Created in-memory cache with unloading after 300s
immich_machine_learning  |                              of inactivity.
immich_machine_learning  | [11/13/24 12:02:02] INFO     Initialized request thread pool with 8 threads.
immich_machine_learning  | [11/13/24 12:02:02] INFO     Application startup complete.
immich_machine_learning  | [11/13/24 12:03:20] ERROR    Worker (pid:10) was sent SIGKILL! Perhaps out of
immich_machine_learning  |                              memory?
immich_machine_learning  | [11/13/24 12:03:20] INFO     Booting worker with pid: 28
immich_redis             | 1:M 13 Nov 2024 17:01:48.618 * Ready to accept connections
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 8
immich_server            | Starting api worker
immich_server            | Starting microservices worker
immich_server            | (node:8) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
immich_server            | (Use `node --trace-deprecation ...` to show where the warning was created)
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 8
➜  docker-compose docker compose logs
immich_machine_learning  | [11/13/24 12:01:50] INFO     Starting gunicorn 23.0.0
immich_redis             | 1:C 13 Nov 2024 17:01:48.615 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
immich_redis             | 1:C 13 Nov 2024 17:01:48.615 # Redis version=6.2.16, bits=64, commit=00000000, modified=0, pid=1, just started
immich_redis             | 1:C 13 Nov 2024 17:01:48.615 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
immich_redis             | 1:M 13 Nov 2024 17:01:48.615 * monotonic clock: POSIX clock_gettime
immich_redis             | 1:M 13 Nov 2024 17:01:48.617 * Running mode=standalone, port=6379.
immich_redis             | 1:M 13 Nov 2024 17:01:48.617 # Server initialized
immich_redis             | 1:M 13 Nov 2024 17:01:48.618 * Ready to accept connections
immich_machine_learning  | [11/13/24 12:01:50] INFO     Listening at: http://[::]:3003 (9)
immich_machine_learning  | [11/13/24 12:01:50] INFO     Using worker: app.config.CustomUvicornWorker
immich_machine_learning  | [11/13/24 12:01:50] INFO     Booting worker with pid: 10
immich_machine_learning  | [11/13/24 12:02:02] INFO     Started server process [10]
immich_machine_learning  | [11/13/24 12:02:02] INFO     Waiting for application startup.
immich_machine_learning  | [11/13/24 12:02:02] INFO     Created in-memory cache with unloading after 300s
immich_machine_learning  |                              of inactivity.
immich_machine_learning  | [11/13/24 12:02:02] INFO     Initialized request thread pool with 8 threads.
immich_machine_learning  | [11/13/24 12:02:02] INFO     Application startup complete.
immich_machine_learning  | [11/13/24 12:03:20] ERROR    Worker (pid:10) was sent SIGKILL! Perhaps out of
immich_machine_learning  |                              memory?
immich_machine_learning  | [11/13/24 12:03:20] INFO     Booting worker with pid: 28
immich_machine_learning  | [11/13/24 12:05:05] ERROR    Worker (pid:28) was sent SIGKILL! Perhaps out of
immich_machine_learning  |                              memory?
immich_machine_learning  | [11/13/24 12:05:06] INFO     Booting worker with pid: 44
immich_machine_learning  | [11/13/24 12:05:51] INFO     Started server process [44]
immich_machine_learning  | [11/13/24 12:05:51] INFO     Waiting for application startup.
immich_machine_learning  | [11/13/24 12:05:51] INFO     Created in-memory cache with unloading after 300s
immich_machine_learning  |                              of inactivity.
immich_machine_learning  | [11/13/24 12:05:51] INFO     Initialized request thread pool with 8 threads.
immich_machine_learning  | [11/13/24 12:05:51] INFO     Application startup complete.
immich_postgres          |
immich_postgres          | PostgreSQL Database directory appears to contain a database; Skipping initialization
immich_postgres          |
immich_postgres          | 2024-11-13 17:01:48.852 UTC [1] LOG:  redirecting log output to logging collector process
immich_postgres          | 2024-11-13 17:01:48.852 UTC [1] HINT:  Future log output will appear in directory "log".
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 8
immich_server            | Starting api worker
immich_server            | Starting microservices worker
immich_server            | (node:8) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
immich_server            | (Use `node --trace-deprecation ...` to show where the warning was created)
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 8
immich_server            | Starting api worker
immich_server            | Starting microservices worker
immich_server            | (node:7) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
immich_server            | (Use `node --trace-deprecation ...` to show where the warning was created)
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 8

Additional information

I see an Error in the log, Worker (pid:10) was sent SIGKILL! Perhaps out of memory? but the system has 16GB of RAM and only less than 10% is utilized.

I also ensured that the UPLOAD_LOCATION and DB_DATA_LOCATION directories all have 775 permission.

[alextran1502]

cc @zackpollard

[hampta]

+1 same problem

log

immich_postgres  |
immich_postgres  | PostgreSQL Database directory appears to contain a database; Skipping initialization
immich_postgres  |
immich_postgres  | 2024-11-14 13:15:34.376 UTC [1] LOG:  redirecting log output to logging collector process
immich_postgres  | 2024-11-14 13:15:34.376 UTC [1] HINT:  Future log output will appear in directory "log".
immich_machine_learning  | [11/14/24 13:17:16] INFO     Starting gunicorn 23.0.0
immich_machine_learning  | [11/14/24 13:17:16] INFO     Listening at: http://[::]:3003 (9)
immich_machine_learning  | [11/14/24 13:17:16] INFO     Using worker: app.config.CustomUvicornWorker
immich_machine_learning  | [11/14/24 13:17:16] INFO     Booting worker with pid: 10
immich_machine_learning  | [11/14/24 13:17:39] INFO     Started server process [10]
immich_machine_learning  | [11/14/24 13:17:39] INFO     Waiting for application startup.
immich_machine_learning  | [11/14/24 13:17:39] INFO     Created in-memory cache with unloading after 300s
immich_machine_learning  |                              of inactivity.
immich_machine_learning  | [11/14/24 13:17:39] INFO     Initialized request thread pool with 4 threads.
immich_machine_learning  | [11/14/24 13:17:39] INFO     Application startup complete.
immich_redis             | 1:C 14 Nov 2024 13:15:32.448 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
immich_redis             | 1:C 14 Nov 2024 13:15:32.448 # Redis version=6.2.16, bits=64, commit=00000000, modified=0, pid=1, just started
immich_redis             | 1:C 14 Nov 2024 13:15:32.448 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
immich_redis             | 1:M 14 Nov 2024 13:15:32.451 * monotonic clock: POSIX clock_gettime
immich_redis             | 1:M 14 Nov 2024 13:15:32.453 * Running mode=standalone, port=6379.
immich_redis             | 1:M 14 Nov 2024 13:15:32.453 # Server initialized
immich_redis             | 1:M 14 Nov 2024 13:15:32.453 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
immich_redis             | 1:M 14 Nov 2024 13:15:32.460 * Ready to accept connections
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4
immich_server            | Initializing Immich v1.120.2
immich_server            | Detected CPU Cores: 4

[alextran1502]

Will it be possible for you guys to increase the CPU Cores and test again? See if it helps

[henryxrl]

I have already increased my core count from 4 to 8. I'm showing you the results after increasing the CPU core count.

[henryxrl]

Previously it was working fine even on a 4-core machine.

[hampta]

how

[alextran1502]

@hampta Oh you are running on a RPi I assume? Pi 4? The resource is not enough to run Immich effectively unfortunately

[CarlvO]

Wait, what? I am running Immich on a Pi4 with 4GB of memory and it has run perfectly fine so far. Will running Immich on a Pi4 not be supported anymore?

[alextran1502]

Hmm, then there is something else going on 🤔

[CarlvO]

I did notice that, after upgradedn to 1.120.2 the system had a high load due to the postgres and node services. I assume it was doing a db dump. If that was indeed the case, it might be related to that. After that, system is running normally.

[thethreeseat]

Hi all, I'm getting the same issue after updating today. I'm running on a QNAP 464 with 32 GB of RAM. I'm seeing immich_microservices and immich_server cycle. I believe the issues I'm seeing are due to changes made in 118 that required additional tweaks on my part. Hopefully I'm right and that will fix it for anyone else hitting that issue. Will report back.

https://github.com/immich-app/immich/releases/tag/v1.118.0

[thethreeseat]

Following those changes decidedly did not work for me. I'm just going to bite the bullet, wipe, reinstall, and be happy my media was not affected.

[philiprenich]

I also was smoothly running on a RPi4 with 8GB of memory. No problems. It looks like the postgres container is restarting constantly?

[philiprenich]

So my issue seems to have come through an upgrade process, but unfortunately I don't know from what version. I have a semi-automated process for updating some docker images and forget this one needed care for breaking changes. I probably upgraded blindly and got the software in to an unrecoverable state. I could have tried reverting versions to see if that would work, but hadn't started using it too much so I did a fresh install and it seems to be working fine.

I do see "error loading image" on previews a fair amount, but that seems to be while jobs are processing for the thumbnails/previews or image recognition.