Slow face detection and smart scan v1.115

[Sithuk]

Has something changed in the new version (v1.115) that slows down "smart search" and "face detection"?

I imported a set of photos that took around 2 days on the previous version. I deleted everything (containers and library) and started again with the new version (v1..115). The smart search and face detection is proceeding incredibly slowly. I've increased the number of concurrent jobs for each from 2 to 4, but it hasn't made much difference.

Has something changed?

[Sithuk]

With an 3.7 Ghz Opteron (E3-1225V5) I am processing around 0.7 photos per minute. That means it will take a month to process 50,000 photos. It took around a da y to carry out face detection on the previous version. Something isn't right. How can I troubleshoot the problem?

I previously added the photos as an external library. This time I imported to Immich's internal library using the CLI. I used the --album tag too to create albums from the folder names. I'm not sure if that would have any influence on the face detection routines.

[bo0tzz]

@mertalev I don't think we have any changes that might cause this right?

[Sithuk]

Update: It is the face detection and smart scan that is running slowly. Face recognition hasn’t started yet. I’ve amended my previous comments to change facial recognition to face detection.

[mertalev]

To clarify, was the previous release 1.114.0? There were no notable changes since then that would introduce a slowdown.

Are there any other jobs besides these two running? Or any assets still being uploaded?

[Sithuk]

Rolling back to v1.114 hasn't fixed the slow progress. Could it be that I have now imported the photos to Immich's internal library rather than scanning them as an external library?

[mertalev]

That could be a lead if library jobs were also running, but not in this case. Have you changed where the thumbs folder is stored? Maybe it's an IO bottleneck.

[Sithuk]

I haven't changed where the thumbs folder is stored. There is very little drive I/O. The system doesn't look I/O constrained.

[Sithuk]

There are quite a few of the following in the immich_server log.

[Nest] 7 - 09/16/2024, 8:15:06 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request to "http://immich-machine-learning:3003" failed with HeadersTimeoutError: Headers Timeout Error
[Nest] 7 - 09/16/2024, 8:15:06 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learning:3003" failed with HeadersTimeoutError: Headers Timeout Error
at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:280:52)
at async /usr/src/app/dist/services/job.service.js:148:36
at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7 - 09/16/2024, 8:15:06 PM ERROR [Microservices:JobService] Object:
{
"id": "bcce9e45-41ff-4be6-870d-278ef2521e11"
}

[mertalev]

Could you share the logs for the machine learning service?

[Sithuk]

sorry for the formatting:

[09/16/24 20:03:49] INFO Application startup compl[29/1883]
[09/16/24 20:04:55] INFO Downloading detection model 'buffalo_l'. This may
take a while.
[09/16/24 20:13:40] INFO Downloading visual model 'ViT-B-32
__openai'. This
may take a while.

[09/16/24 20:13:40] WARNING Failed to load detection model 'bu
ffalo_l'.
Clearing cache.

[09/16/24 20:13:40] WARNING Attempted to clear cache for model
'buffalo_l', but
cache directory does not exist

[09/16/24 20:22:24] INFO Downloading visual model 'ViT-B-32
__openai'. This
may take a while.

[09/16/24 20:22:24] WARNING Failed to load visual model 'ViT-B
-32__openai'.
Clearing cache.

[09/16/24 20:22:24] WARNING Attempted to clear cache for model

                         'ViT-B-32__openai', but cache dire

ctory does not �������
exist

[09/16/24 20:31:09] INFO Downloading detection model 'buffa
lo_l'. This may take a while.
[09/16/24 20:31:09] WARNING Failed to load visual model 'ViT-B
-32__openai'. Clearing cache.
[09/16/24 20:31:09] WARNING Attempted to clear cache for model
'ViT-B-32__openai', but cache dire
ctory does not exist
[09/16/24 20:39:53] INFO Downloading detection model 'buffa
lo_l'. This may take a while.
[09/16/24 20:39:53] WARNING Failed to load detection model 'bu
ffalo_l'. Clearing cache.
[09/16/24 20:39:53] WARNING Attempted to clear cache for model
'buffalo_l', but cache directory does not exist

[mertalev]

Can you stop the containers, delete the machine learning container, then delete the model-cache volume? This is a bug with model downloading that I haven't root-caused yet.

[Sithuk]

I have stopped the containers, deleted the machine learning container, and deleted the model-cache volume. I then did "sudo docker compose up -d". The issue is still present.

I tried a quick connectivity check by opening a shell within the machine learning container and trying to ping google.com. But ping isn't installed. Is there a way for me to try to manually download the file from within the container? I didn't check whether wget or curl were installed, but if they are then I could try a manual download?

[mertalev]

If you're using Docker Compose and the container is running, you can use the following commands to manually download these models within the container:

docker exec \
	-e MODEL_NAME=ViT-B-32__openai \
	-e MODEL_TASK=clip \
	immich_machine_learning sh -c 'huggingface-cli download "immich-app/$MODEL_NAME" --local-dir "/cache/$MODEL_TASK/$MODEL_NAME" --cache-dir "/cache/$MODEL_TASK/$MODEL_NAME"'

docker exec \
	-e MODEL_NAME=buffalo_l \
	-e MODEL_TASK=facial-recognition \
	immich_machine_learning sh -c 'huggingface-cli download "immich-app/$MODEL_NAME" --local-dir "/cache/$MODEL_TASK/$MODEL_NAME" --cache-dir "/cache/$MODEL_TASK/$MODEL_NAME"'

[Sithuk]

Thank you for your help mertalev. I gave up trying to fix the issue in the end. I run the Immich docker inside an lxc container (using lxd) with nginx. I have deleted and re-created the lxc container and everything is working again, including the face detection and smart scan.