Unepexected error - Immich no longer functions.

[cemkur]

I am running an older version of immich v1.117 (no matter what I did, I could not get the newer version to work as the immich_server container stuck in restart loop)

The only thing I did recently was the Supporter pledge (which is unrelated I know) after which point I haven not been able to open any of the images. I get the following error.

Error - Something went wrong

Failed to get asset info (500)

Stacktrace
Error: Error: 500
at Object.he [as ok] (https://immich.mydomain.com/_app/immutable/chunks/fetch-client.DFLUroz2.js:1:2872)
at async qt (https://immich.mydomain.com/_app/immutable/nodes/17.B9fYI29D.js:1:2358)
at async Le (https://immich.mydomain.com/_app/immutable/chunks/entry.DZ7M39Ac.js:1:14646)
Get Help

Read Changelog

Check Logs

Bottom of the immich_server logs shows the following

[Nest] 7 - 12/08/2024, 12:00:00 AM ERROR [Microservices:JobService] Unable to run job handler (backgroundTask/asset-deletion): QueryFailedError: could not open file "base/16384/17336": No such file or directory
[Nest] 7 - 12/08/2024, 12:00:00 AM ERROR [Microservices:JobService] QueryFailedError: could not open file "base/16384/17336": No such file or directory
at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async DeleteQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/DeleteQueryBuilder.js:52:33)
at async SubjectExecutor.executeRemoveOperations (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:461:17)
at async SubjectExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:104:9)
at async EntityPersistExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/EntityPersistExecutor.js:140:21)
at async AssetRepository.remove (/usr/src/app/dist/repositories/asset.repository.js:202:9)
at async AssetService.handleAssetDeletion (/usr/src/app/dist/services/asset.service.js:187:9)
at async /usr/src/app/dist/services/job.service.js:163:36
at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
[Nest] 7 - 12/08/2024, 12:00:00 AM ERROR [Microservices:JobService] Object:

[Nest] 31 - 12/21/2024, 1:32:59 PM LOG [Api:RouterExplorer] Mapped {/api/view/folder, GET} route
[Nest] 31 - 12/21/2024, 1:32:59 PM LOG [Api:NestApplication] Nest application successfully started
[Nest] 31 - 12/21/2024, 1:32:59 PM LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:3001 [v1.117.0] [PRODUCTION]
[Nest] 31 - 12/21/2024, 1:33:00 PM LOG [Api:EventRepository] Websocket Connect: qpeS5_JA1hscB6S6AAAB
[Nest] 31 - 12/21/2024, 1:35:18 PM LOG [Api:EventRepository] Websocket Disconnect: qpeS5_JA1hscB6S6AAAB
[Nest] 31 - 12/21/2024, 1:35:20 PM LOG [Api:EventRepository] Websocket Connect: vYPgdRqV6Q4YVO1pAAAD
[Nest] 31 - 12/21/2024, 1:35:22 PM ERROR [Api:ErrorInterceptor~roc6qlom] Database error: QueryFailedError: pg_attribute catalog is missing 6 attribute(s) for relation OID 16550

I really don't want to go through the installation steps etc. Does anyone have any idea about how to address this?

Thank you

ck

[mmomjian]

Your database seems corrupted. Nothing to do with supporter, no. Do you have database backups?

[mmomjian]

Full VM backups are notorious for causing issues for databases since there is no guarantee of consistency across the backup if the VM is running. This is why our docs require a pg_dumpall backup

[cemkur]

Well, I have learned my lesson for sure now. Would the full VM backups work if I stop the immich and db containers beforehand?

[cemkur]

BTW Proxmox BPS full backups are only incremental backups, despite the name.

[mmomjian]

When I see full VM I mean block/file level. It doesn’t matter whether it’s stored incrementally or not (and proxmox backups are technically full backups that are deduplicated. Incremental backups require a base image and then “fast forwarded” through the increments)

if the(immich_postgres specifically) docker container is stopped, it would be a lot more safe. However database dumps (which are built in with version 1.120) are still safest.

[cemkur]

If I can't get anywhere with the restore, I will build a new immich and import the media using the immich-go toool unless you have another recommendation. My only concern is that there are four users and each had their own favorites, which I assume will be lost.