how to support new models like qwen2.5 vl?

[lucasjinreal]

how to support new models like qwen2.5 vl? Any step by step tutorial for this?

[edgarriba]

Probably we need to implement some adapters for that. We were planning to support a second family of models for SmolVLM soon but happy also to support the qwen fam. Are you aware of any rust native implementation? I checked in candle but seems it doesn't support vision?

[lucasjinreal]

I noticed that there are some PRs added to support Qwen2.5 VL in candle, it's Rust native.

However, I'm not sure if they're gonna support quantization or not.

In terms of model support, based on candle, separating some model inference as a crate is good, not based only on candle, since they actually rarely add new models. Many new model pull requests were ignored and they just don't merge them.

[edgarriba]

That’s why we are starting with the interns this separated crate for inference only https://github.com/kornia/kornia-rs/tree/main/crates/kornia-vlm

We’ll add first Paligemma and SmolVLM. We are open to explore alternatives to Candle too (and considering to build ourselves if we get the right resources).

If you are open to, we could collaborate in this space.

[lucasjinreal]

Kornia-VLM is awesome!
We can start porting paligemma and smolvlm, and qwen2.5 vl could also be considered.

I was previously integrating some model inference in this crate: https://github.com/lucasjinreal/Crane though.
We are focusing on supporting more diverse models such as Qwen3, VL, Spark-TTS etc.
However, due to my limited spare time, the project progress is slow.

I wish I could see more support from the kornia team!