How to complie Candle for Native devices (Android, IOS)?

[akashicMarga]

How can we build candle binaries which can be used in Android or IOS projects?

[soupslurpr]

I used UniFFI to generate Kotlin bindings to my Rust project and then call the necessary methods from Kotlin.
Additionally, in .cargo/config.toml I put:

[target.aarch64-linux-android]
ar = "C:/Users/user/AppData/Local/Android/Sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/windows-x86_64/bin/llvm-ar.exe"
linker = "C:/Users/user/AppData/Local/Android/Sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/windows-x86_64/bin/aarch64-linux-android23-clang.cmd"

[target.x86_64-linux-android]
ar = "C:/Users/user/AppData/Local/Android/Sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/windows-x86_64/bin/llvm-ar.exe"
linker = "C:/Users/user/AppData/Local/Android/Sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/windows-x86_64/bin/x86_64-linux-android23-clang.cmd"

[env]
CC_aarch64-linux-android = "C:/Users/user/AppData/Local/Android/Sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/windows-x86_64/bin/aarch64-linux-android23-clang.cmd"
CC_x86_64-linux-android = "C:/Users/user/AppData/Local/Android/Sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/windows-x86_64/bin/x86_64-linux-android23-clang.cmd"
AR_aarch64_linux_android = "C:/Users/user/AppData/Local/Android/Sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/windows-x86_64/bin/llvm-ar.exe"
AR_x86_64_linux_android = "C:/Users/user/AppData/Local/Android/Sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/windows-x86_64/bin/llvm-ar.exe"

Adjust for where the ndk is for you and the installed version.

Finally, I also added tokenizers with default-features = false and features = ["onig"] so OpenSSL isn't compiled.

[soupslurpr]

Are you using the quantized model with llama.cpp maybe?

[akashicMarga]

Are you using the quantized model with llama.cpp maybe?

Yes

[soupslurpr]

And you're running the quantized one with Candle too?

[akashicMarga]

And you're running the quantized one with Candle too?

Yes same 4 bit quantised.

[soupslurpr]

Ah I think it's just that Candle doesn't have all the optimizations https://github.com/huggingface/candle/blob/cfab6e761696c18b1ce5d3a339ab57ef191ca749/candle-core/src/quantized/neon.rs