How to get acceleration on mali-G610 (orange pi 5) #11056

plord12 · 2024-07-12T14:46:18Z

plord12
Jul 12, 2024

Hi All,

Now that #11019 is resolved, anyone got acceleration working with maili GPU ?

Firstly I found it hard to find a definitive source for libmali ... eventually I used the deb packages at https://github.com/tsukumijima/libmali-rockchip/releases ( I chose libmali-valhall-g610-g6p0-gbm_1.9-1_arm64.deb ).

I updated hwaccel.ml.yml :

services:
  armnn:
    devices:
      - /dev/mali0:/dev/mali0
    volumes:
      - /lib/firmware/mali_csffw.bin:/lib/firmware/mali_csffw.bin:ro # Mali firmware for your chipset (not always required depending on the driver)
      - /usr/lib/aarch64-linux-gnu/libmali.so.1.9.0:/usr/lib/libmali.so:ro # Mali driver for your chipset (always required)

and docker-compose.yml :

  immich-machine-learning:
   container_name: immich_machine_learning
   # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
   # Example tag: ${IMMICH_VERSION:-release}-cuda
   image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-armnn
   extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
     file: hwaccel.ml.yml
     service: armnn # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable

Now when I click on the smart search job I see :

$ docker logs -f immich_machine_learning
...
[07/12/24 14:32:52] DEBUG    Setting model format to armnn                      
[07/12/24 14:32:52] INFO     Loading visual model 'ViT-B-32__openai' to memory  
[07/12/24 14:32:52] DEBUG    Loading visual preprocessing config for CLIP model 
                            'ViT-B-32__openai'                                 
[07/12/24 14:32:52] DEBUG    Loaded visual preprocessing config for CLIP model  
                            'ViT-B-32__openai'                                 
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
[07/12/24 14:32:53] INFO     Loading ANN model                                  
                            /cache/clip/ViT-B-32__openai/visual/model.armnn ...
[07/12/24 14:32:54] INFO     Loaded ANN model with ID 0                         
[07/12/24 14:33:00] DEBUG    Checking for inactivity...                         
[07/12/24 14:33:10] DEBUG    Checking for inactivity...                         
[07/12/24 14:33:20] DEBUG    Checking for inactivity...                         
[07/12/24 14:33:30] DEBUG    Checking for inactivity...                         
[07/12/24 14:33:40] DEBUG    Checking for inactivity...                         
[07/12/24 14:33:50] DEBUG    Checking for inactivity...                         
[07/12/24 14:34:00] DEBUG    Checking for inactivity...                         
[07/12/24 14:34:10] DEBUG    Checking for inactivity...                         
[07/12/24 14:34:20] DEBUG    Checking for inactivity...                         
[07/12/24 14:34:30] DEBUG    Checking for inactivity...                         
[07/12/24 14:34:39] ERROR    Worker (pid:10) was sent code 139!        
...

So it seems libmali is being picked up. But not much info on the failure.

On the host I see :

# dmesg -w
....
[ 1120.001326] mali fb000000.gpu: Power transition timed out unexpectedly
[ 1120.001360] mali fb000000.gpu: 	MCU desired = 1
[ 1120.001378] mali fb000000.gpu: 	MCU sw state = 2
[ 1120.001394] mali fb000000.gpu: Current state :
[ 1120.001415] mali fb000000.gpu: 	Shader=0000000000000000
[ 1120.001433] mali fb000000.gpu: 	Tiler =0000000000000000
[ 1120.001450] mali fb000000.gpu: 	L2    =0000000000000001
[ 1120.001467] mali fb000000.gpu: 	MCU status = 2
[ 1120.001484] mali fb000000.gpu: Cores transitioning :
[ 1120.001502] mali fb000000.gpu: 	Shader=0000000000000000
[ 1120.001520] mali fb000000.gpu: 	Tiler =0000000000000000
[ 1120.001538] mali fb000000.gpu: 	L2    =0000000000000000
[ 1120.001555] mali fb000000.gpu: Sending reset to GPU - all running jobs will be lost
[ 1120.001581] mali fb000000.gpu: Preparing to soft-reset GPU
[ 1120.001610] mali fb000000.gpu: Wait for MCU power on failed on scheduling tick/tock
[ 1120.002221] mali fb000000.gpu: Resetting GPU (allowing up to 500 ms)
[ 1120.002243] mali fb000000.gpu: Register state:
[ 1120.002269] mali fb000000.gpu:   GPU_IRQ_RAWSTAT=0x00000000   GPU_STATUS=0x00000000  MCU_STATUS=0x00000002
[ 1120.002294] mali fb000000.gpu:   JOB_IRQ_RAWSTAT=0x00000000   MMU_IRQ_RAWSTAT=0x00000000   GPU_FAULTSTATUS=0x00000000
[ 1120.002318] mali fb000000.gpu:   GPU_IRQ_MASK=0x00000000   JOB_IRQ_MASK=0x00000000   MMU_IRQ_MASK=0x00000000
[ 1120.002339] mali fb000000.gpu:   PWR_OVERRIDE0=0x00000000   PWR_OVERRIDE1=0x00000000
[ 1120.002360] mali fb000000.gpu:   SHADER_CONFIG=0x00000000   L2_MMU_CONFIG=0x00000000   TILER_CONFIG=0x00000000
[ 1120.002390] mali fb000000.gpu: Firmware trace buffer dump:
[ 1120.002962] mali fb000000.gpu: reloading firmware
[ 1120.003605] mali fb000000.gpu: Reset complete
[ 1122.564515] mali fb000000.gpu: Power transition timed out unexpectedly
[ 1122.564540] mali fb000000.gpu: 	MCU desired = 1
[ 1122.564551] mali fb000000.gpu: 	MCU sw state = 2
[ 1122.564563] mali fb000000.gpu: Current state :
[ 1122.564578] mali fb000000.gpu: 	Shader=0000000000000000
[ 1122.564591] mali fb000000.gpu: 	Tiler =0000000000000000
[ 1122.564602] mali fb000000.gpu: 	L2    =0000000000000001
[ 1122.564614] mali fb000000.gpu: 	MCU status = 2
[ 1122.564624] mali fb000000.gpu: Cores transitioning :
[ 1122.564636] mali fb000000.gpu: 	Shader=0000000000000000
[ 1122.564647] mali fb000000.gpu: 	Tiler =0000000000000000
[ 1122.564661] mali fb000000.gpu: 	L2    =0000000000000000
[ 1122.564671] mali fb000000.gpu: Sending reset to GPU - all running jobs will be lost
[ 1122.564686] mali fb000000.gpu: Preparing to soft-reset GPU
[ 1122.564722] mali fb000000.gpu: Wait for MCU power on failed on scheduling tick/tock
[ 1122.564995] mali fb000000.gpu: Resetting GPU (allowing up to 500 ms)
[ 1122.565008] mali fb000000.gpu: Register state:
[ 1122.565024] mali fb000000.gpu:   GPU_IRQ_RAWSTAT=0x00000000   GPU_STATUS=0x00000000  MCU_STATUS=0x00000002
[ 1122.565039] mali fb000000.gpu:   JOB_IRQ_RAWSTAT=0x00000000   MMU_IRQ_RAWSTAT=0x00000000   GPU_FAULTSTATUS=0x00000000
[ 1122.565054] mali fb000000.gpu:   GPU_IRQ_MASK=0x00000000   JOB_IRQ_MASK=0x00000000   MMU_IRQ_MASK=0x00000000
[ 1122.565068] mali fb000000.gpu:   PWR_OVERRIDE0=0x00000000   PWR_OVERRIDE1=0x00000000
[ 1122.565082] mali fb000000.gpu:   SHADER_CONFIG=0x00000000   L2_MMU_CONFIG=0x00000000   TILER_CONFIG=0x00000000
[ 1122.565100] mali fb000000.gpu: Firmware trace buffer dump:
[ 1122.565320] mali fb000000.gpu: reloading firmware
[ 1122.565932] mali fb000000.gpu: Reset complete

I've no clue what this means.

Thanks!

smiggiddy · 2024-08-23T02:24:07Z

smiggiddy
Aug 23, 2024

did you figure this out?

1 reply

plord12 Aug 26, 2024
Author

When searching photos I see -

                             FileNotFoundError: Model file not found:           
                             /cache/clip/ViT-B-32__openai/textual/model.armnn   
[08/26/24 07:14:55] WARNING  ARMNN is available, but model 'ViT-B-32__openai'   
                             does not support it.

I guess this is still pending.

jacopomaroli · 2024-09-11T16:29:06Z

jacopomaroli
Sep 11, 2024

it seems to work for me:

[09/11/24 17:19:17] INFO     Starting gunicorn 23.0.0                           
[09/11/24 17:19:17] INFO     Listening at: http://[::]:3003 (9)                 
[09/11/24 17:19:17] INFO     Using worker: app.config.CustomUvicornWorker       
[09/11/24 17:19:17] INFO     Booting worker with pid: 10                        
[09/11/24 17:19:23] INFO     Started server process [10]                        
[09/11/24 17:19:24] INFO     Waiting for application startup.                   
[09/11/24 17:19:24] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[09/11/24 17:19:24] INFO     Initialized request thread pool with 8 threads.    
[09/11/24 17:19:24] DEBUG    Checking for inactivity...                         
[09/11/24 17:19:24] INFO     Application startup complete.                      
[09/11/24 17:19:31] DEBUG    Setting model format to armnn                      
[09/11/24 17:19:31] INFO     Loading visual model 'ViT-B-32__openai' to memory  
[09/11/24 17:19:31] DEBUG    Loading visual preprocessing config for CLIP model 
                             'ViT-B-32__openai'                                 
[09/11/24 17:19:31] DEBUG    Loaded visual preprocessing config for CLIP model  
                             'ViT-B-32__openai'                                 
arm_release_ver: g13p0-01eac0, rk_so_ver: 10
[09/11/24 17:19:31] INFO     Loading ANN model                                  
                             /cache/clip/ViT-B-32__openai/visual/model.armnn ...
[09/11/24 17:19:32] INFO     Loaded ANN model with ID 0

problem is I get load on my CPU and running

sudo watch -n 1 cat /sys/kernel/debug/rknpu/load

or even

sudo watch -n 1 cat /sys/kernel/debug/rkrga/load

shows no activty. how do I confirm it's actually using HW accel?

my setup: orangepi 5 Joshua-Riek ubuntu

orangepi@opi5:~$ uname -r
5.10.0-1012-rockchip
orangepi@opi5:~$ uname -a
Linux opi5 5.10.0-1012-rockchip #12-Ubuntu SMP Wed Aug 14 22:22:22 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
orangepi@opi5:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

I downloaded and installed libmali-valhall-g610-g13p0-gbm_1.9-1_arm64.deb from the repo in the first comment.
the only change I had to do was mounting both libmali.so and libmali.so.1 into my container.
this distro also includes a libmali-valhall-g610-g13p0-x11-wayland-gbm.so but mounting that as libmali.so will trigger an infinite amount of other requirements for other *.so files mostly related to X, so I pulled the lean non-wayland library I mentioner earlier

0 replies

Mahoo12138 · 2024-10-14T05:28:26Z

Mahoo12138
Oct 14, 2024

Look forward to any progress from you.

3 replies

LeoRX Nov 15, 2024

it took some time for me to get things going, but I'm also seeing the "Power transition timed out unexpectedly" issue. :(

forgot to mention I'm running rock pi 5a

mertalev Nov 15, 2024
Maintainer

Rockchip boards tend to be sensitive to the power block and cable you use. I had to try like 4 different things before I found something that worked reliably. Check what others have used with success for your model. A "dumb", non-PD charger with the right voltage and amperage is more likely to work well.

LeoRX Nov 28, 2024

@mertalev Thank you for your suggestion. Unfortunately I can't find the power supply and cable that suits. tried a few high power rated cable and various power supply, including the Radxa 30w power adapter.

LeoRX · 2024-11-28T22:35:16Z

LeoRX
Nov 28, 2024

Finally! after another few hours of fiddle around and got this working.
I can't comment why "Power transition timed out unexpectedly" happens, but it is most likely due to missing... something. I had cli version of the Radxa OS and ended up install the KDE version, everything started to work afterwards.

4 replies

RikudouGoku Dec 2, 2024

Do you mind giving a step by step guide on how you did this? I just installed Immich on my Orange Pi 5 Plus so looking to try it out with hardware acceleration.

LeoRX Dec 2, 2024

@RikudouGoku have you follow the steps in the doc? https://immich.app/docs/features/ml-hardware-acceleration
are you seeing errors?

btw.. there was an update of driver a few hours ago at https://github.com/tsukumijima/libmali-rockchip/releases

LeoRX Dec 2, 2024

Also want to note, I have not able to get any clip model working except for the default "ViT-B-32__openai"...

RikudouGoku Dec 2, 2024

I did check the docs. "/dev/mali0", "/usr/lib/libmali.so" and "/lib/firmware/mali_csffw.bin" exists.

Here is how my docker-compose.yml looks under the ml area:

immich-machine-learning:
container_name: immich_machine_learning
# For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
# Example tag: ${IMMICH_VERSION:-release}-cuda
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-armnn
extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
file: hwaccel.ml.yml
service: armnn # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the -wsl version for WSL2 where applicable
volumes:
- model-cache:/cache
env_file:
- .env
restart: always
healthcheck:
disable: false

plord12 · 2024-12-02T08:48:03Z

plord12
Dec 2, 2024
Author

I have upgraded to libmali-valhall-g610-g24p0-gbm_1.9-1_arm64.deb and have refresh the model cache.

When running text searches I see -

[12/02/24 08:38:12] INFO     Downloading textual model 'ViT-B-32__openai'. This 
                             may take a while.                                  
Fetching 14 files: 100%|██████████| 14/14 [00:00<00:00, 1542.96it/s]
[12/02/24 08:38:12] INFO     Loading textual model 'ViT-B-32__openai' to memory 
[12/02/24 08:38:12] ERROR    Model file not found:                              
                             /cache/clip/ViT-B-32__openai/textual/model.armnn

When forcing a facial recognition, I see -

[12/02/24 08:42:55] INFO     Loading detection model 'buffalo_l' to memory      
arm_release_ver: g24p0-00eac0, rk_so_ver: 4
[12/02/24 08:42:55] INFO     Loading ANN model                                  
                             /cache/facial-recognition/buffalo_l/detection/model
                             .armnn ...                                         
Binary version (37) differs from expected (44)
terminate called after throwing an instance of 'cl::Error'
  what():  clCreateProgramWithBinary

I guess I'll try with the 'g13p0' version

2 replies

plord12 Dec 2, 2024
Author

I have upgraded to libmali-valhall-g610-g24p0-gbm_1.9-1_arm64.deb and have refresh the model cache.

Ah. I need the g6p0 version (presumably some binary compatibility).

Specifically, I'm currently trying https://github.com/tsukumijima/libmali-rockchip/releases/download/v1.9-1-3238416/libmali-valhall-g610-g6p0-gbm_1.9-1_arm64.deb

I'm not seeing errors with smart search admin job - clip model appears to be loaded and used. Accelerated text searches not supported.

dmesg is still very noisy ... GPU reset many times. I've no idea if this stops acceleration being used.

LeoRX Dec 3, 2024

I haven't seen similar errors. However, straight after the server start, you can run dmesg | grep gpu to find the version of the user space driver you should be using.

mine looks like this

[ 11.819838] mali fb000000.gpu-mali: Kernel DDK version g21p0-01eac0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to get acceleration on mali-G610 (orange pi 5) #11056

{{title}}

Replies: 5 comments 10 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

How to get acceleration on mali-G610 (orange pi 5) #11056

Replies: 5 comments · 10 replies

plord12 Aug 26, 2024 Author

mertalev Nov 15, 2024 Maintainer

plord12 Dec 2, 2024 Author

plord12 Dec 2, 2024 Author

Replies: 5 comments 10 replies

plord12 Aug 26, 2024
Author

mertalev Nov 15, 2024
Maintainer

plord12
Dec 2, 2024
Author

plord12 Dec 2, 2024
Author