[Feature] Machine Learning using Rockchip NPU on supported boards.

[yoni13]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

Since immich already supports Hardware Transcoding with hardware acceleration via rkmpp, maybe we can make machine learning working on rockchip NPUs.

Links:
rknn-toolkit2
frigate's implantation

Platform

• Server
• Web
• Mobile

[yoni13]

Update: I am able to run "ViT-B-32__openai" using rknn-toolkit2 on emulator, I haven't verify the accuracy but I will say it got some progress.

[yoni13]

Those methods are mainly there for ONNX Runtime. If the RKNN engine doesn't need to know the input/output names, then it's okay to not implement those methods / just return a dummy. There are workarounds we could do but if we don't need this functionality then it's kind of moot.

From the RetinaFace Example in RKNN Model Zoo, I don't see any variables related to input/output names (or maybe I missed something but hope not).

If I want to try to make it return a dummy value, would you recommend what value should I use?

[mertalev]

AnnSession uses None for the name and only provides the shape, so this can do the same (or provide None for both if the shape is also unavailable).

[yoni13]

I am out of ideas now 🫠.
I think I need to know more about insightface to solve this.

[mertalev]

Can you clarify what the issue is? Is it that the InsightFace models don't work with RKNN, or is it that you're not sure how to handle the get_inputs / get_outputs interface?

[yoni13]

Can you clarify what the issue is? Is it that the InsightFace models don't work with RKNN, or is it that you're not sure how to handle the get_inputs / get_outputs interface?

I am not sure how to handle the get_inputs and get_outputs, the example in rknn model zoo just rknn.inference(inputs=[infer_img]) and yes I don't know how to handle that😐

[yoni13]

(machine-learning-py3.11) root@orangepi3b:~/immich/machine-learning# ./start.sh 
ERROR: ld.so: object '/usr/lib/aarch64-linux-gnu/libmimalloc.so.2' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
[12/03/24 12:15:23] INFO     Starting gunicorn 23.0.0                                                                                                                                        
[12/03/24 12:15:23] INFO     Listening at: http://[::]:3003 (5620)                                                                                                                           
[12/03/24 12:15:23] INFO     Using worker: app.config.CustomUvicornWorker                                                                                                                    
[12/03/24 12:15:23] INFO     Booting worker with pid: 5621                                                                                                                                   
[12/03/24 12:15:32] I        Started server process [5621]                                                                                                                                   
[12/03/24 12:15:32] I        Waiting for application startup.                                                                                                                                
[12/03/24 12:15:32] I        Created in-memory cache with unloading after 300s of inactivity.                                                                                                
[12/03/24 12:15:32] I        Initialized request thread pool with 4 threads.                                                                                                                 
[12/03/24 12:15:32] I        Application startup complete.                                                                                                                                   
[12/03/24 12:15:34] I        Loading textual model 'ViT-B-32__openai' to memory                                                                                                              
W rknn-toolkit-lite2 version: 2.3.0
[12/03/24 12:15:34] I        Loading RKNN model from /cache/clip/ViT-B-32__openai/textual/model.rknn                                                                                         
--> Load RKNN model
I RKNN: [12:15:35.403] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [12:15:35.403] RKNN Driver Information, version: 0.9.7
I RKNN: [12:15:35.404] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU lite, target platform: rk3566, framework name: ONNX, framework layout: NCHW, model inference type: static_shape
W RKNN: [12:15:35.728] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.

Can someone help me with dynamic_shapes&static_shapes? I have no idea what are them.
Or we can ignore that warning?

[mertalev]

If it is a static shape RKNN model, please ignore the above warning message

I think you can ignore it if there are no errors.

[yoni13]

Small update: I tested the model in Immich, and the ViT-B-32 shows a latency difference of 200 ms after using RKNN(from cpu bcs I am on dietpi).

[yoni13]

[Nest] 7  - 12/22/2024, 11:32:29 PM   ERROR [Microservices:JobService] QueryFailedError: pgvecto.rs: Bad literal.
INFORMATION: hint = Expect a number.
at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
at async SearchRepository.upsert (/usr/src/app/dist/repositories/search.repository.js:210:9)
at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/services/smart-info.service.js:109:9)
at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:148:28)
at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:134:13)
at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)

But rerun smart search still doesn't work