Idefics2 inference

[dancixx]

Hi guys,

I tried to use idefics2 as a multimodal model, but I always got a weird answer, please specify the level of the exercise, did somebody same issue in case of using this model?

I followed the mistral.rs examples for the inference.

[EricLBuehler]

Hi @dancixx I'll take a look 😄

[EricLBuehler]

@dancixx

I ran:

cargo run --features cuda -- --port 1234 --no-paged-attn --isq Q4K vision-plain -m HuggingFaceM4/idefics2-8b-chatty -a idefics2

And then:

python3 examples/server/idefics2.py

Which gave:

The image depicts a group of orange ants climbing over a black pole. The ants are moving in the same direction, forming a line as they ascend the pole.

What is the result you are seeing?

[dancixx]

@EricLBuehler This is the rust code that Im using.

use either::Either;
use image::ImageReader;
use indexmap::IndexMap;
use std::sync::Arc;
use tokio::sync::mpsc::channel;

use mistralrs::{
    Constraint, DefaultSchedulerMethod, Device, DeviceMapMetadata, MistralRs, MistralRsBuilder,
    ModelDType, NormalRequest, Request, RequestMessage, Response, Result, SamplingParams,
    SchedulerConfig, TokenSource, VisionLoaderBuilder, VisionLoaderType, VisionSpecificConfig,
};

/// Gets the best device, cpu, cuda if compiled with CUDA
pub(crate) fn best_device() -> Result<Device> {
    #[cfg(not(feature = "metal"))]
    {
        Device::cuda_if_available(0)
    }
    #[cfg(feature = "metal")]
    {
        Device::new_metal(0)
    }
}

fn setup() -> anyhow::Result<Arc<MistralRs>> {
    // Select a Mistral model
    let loader = VisionLoaderBuilder::new(
        VisionSpecificConfig {
            use_flash_attn: false,
            prompt_batchsize: None,
        },
        None,
        None,
        Some("HuggingFaceM4/idefics2-8b".to_string()),
    )
    .build(VisionLoaderType::Idefics2);
    // Load, into a Pipeline
    let pipeline = loader.load_model_from_hf(
        None,
        TokenSource::CacheToken,
        &ModelDType::Auto,
        &best_device()?,
        false,
        DeviceMapMetadata::dummy(),
        None,
        None, // No PagedAttention.
    )?;
    // Create the MistralRs, which is a runner
    Ok(MistralRsBuilder::new(
        pipeline,
        SchedulerConfig::DefaultScheduler {
            method: DefaultSchedulerMethod::Fixed(5.try_into().unwrap()),
        },
    )
    .build())
}

fn main() -> anyhow::Result<()> {
    let mistralrs = setup()?;

    let (tx, mut rx) = channel(10_000);
    let image = ImageReader::open("demo14.png")?.decode()?;
    let request = Request::Normal(NormalRequest {
        messages: RequestMessage::VisionChat {
            images: vec![image],
            messages: vec![IndexMap::from([
                ("role".to_string(), Either::Left("user".to_string())),
                (
                    "content".to_string(),
                    Either::Left(String::from("<|image_1|>\nWhat is the license plate number in the picture. The answer has to be in JSON format like {\"license_plate_number\": \"ABC123\"}.")),
                ),
            ])],
        },
        sampling_params: SamplingParams::default(),
        response: tx,
        return_logprobs: false,
        is_streaming: false,
        id: 0,
        constraint: Constraint::None,
        suffix: None,
        adapters: None,
        tools: None,
        tool_choice: None,

    });
    mistralrs.get_sender()?.blocking_send(request)?;

    let response = rx.blocking_recv().unwrap();
    match response {
        Response::Done(c) => println!(
            "Text: {}, Prompt T/s: {}, Completion T/s: {}",
            c.choices[0].message.content.as_ref().unwrap(),
            c.usage.avg_prompt_tok_per_sec,
            c.usage.avg_compl_tok_per_sec
        ),
        Response::InternalError(e) => panic!("Internal error: {e}"),
        Response::ValidationError(e) => panic!("Validation error: {e}"),
        Response::ModelError(e, c) => panic!(
            "Model error: {e}. Response: Text: {}, Prompt T/s: {}, Completion T/s: {}",
            c.choices[0].message.content.as_ref().unwrap(),
            c.usage.avg_prompt_tok_per_sec,
            c.usage.avg_compl_tok_per_sec
        ),
        _ => unreachable!(),
    }
    Ok(())
}

This is the answer what Im always getting: (always the same)
Text: Speciify an examination/test you took for which the test/exam is a final evaluation at the end of the school term.

To determine follow-up responses for the prompts provided, we need specific information about the examination or test you took. Please provide more context and details about the particular examination/test that you took.

Also, specify the level of education from which the test belongs (e.g., high school, undergraduate, graduate, professional, etc.). This will help to guide the follow-up answers in a more focused manner., Prompt T/s: 0.3465724, Completion T/s: 2.6231098

[EricLBuehler]

@dancixx I see! Yes, this was a mistake on my part, sorry about that. #698 was just merged and should fixed this, you can check the diff to see what you need to change (it's just the messages as the fix).

[dancixx]

@EricLBuehler thank you for your help. I will test is a bit later today. :)

[dancixx]

@EricLBuehler Thanks, its working. :)

[EricLBuehler]

@dancixx glad I could help!