Using uqff downloads the full safetensors weights.

[chigkim]

Describe the bug

I ran the following, and it downloaded both uqff as well as full weights from Meta. I tried to skip -m, but it seems -m is required.

./mistralrs-server -i vision-plain -m meta-llama/Llama-3.2-11B-Vision-Instruct -a vllama --from-uqff EricB/Llama-3.2-11B-Vision-Instruct-UQFF/llama-3.2-11b-vision-hqq8.uqff

Latest commit or version

3e79d85

[vietvudanh]

I had same problem.

Not sure if I read rust code correctly but it seems Args in mistralrs-server/src/main.rs has no option for uqff. Only has model, which loads in this case meta-llama/Llama-3.2-11B-Vision-Instruct, so full tensor of base model.

mistral.rs/mistralrs-server/src/main.rs

Line 53 in 3957283

struct Args {

[EricLBuehler]

@chigkim @vietvudanh you can now load UQFF models without downloading the full weights in #849!

For example (https://huggingface.co/EricB/Llama-3.2-11B-Vision-Instruct-UQFF):

./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-q4k.uqff

More models can be found here: https://huggingface.co/collections/EricB/uqff-670e4a49d56ecdd3f7f0fd4c.