support llava_onevision

[defaultak01]

No description provided.

[defaultak01]

@Yangsenqiao First of all, this is an excellent piece of work, and I really appreciate you making the code publicly available.

I noticed that in your demonstration of how VisionZip improves video understanding speed, you used LLAVA_OneVision. However, I encountered some compatibility issues when deploying it locally. For example:

1. The prepare_inputs_labels_for_multimodal_visionzip function does not accept the **modalities** parameter as input, whereas the corresponding function in LLAVA includes this parameter.
2. LLAVA_OneVision uses the SigLIP vision encoder (corresponding to **SigLipVisionTower**), but the code you provided is only adapted for **CLIPVisionTower**.

Could you please share the code related to LLAVA_OneVision that you used in the demonstration?

[effortprogrammer]

+1

[Yangsenqiao]

Please refer to my issue #9. Thanks a lot. 😊

[liyucheng09]

@Yangsenqiao Hi Senqiao, can we simply replace CLIPVisionTower with SigLipVisionTower in the pattch function? will this reproduce similar results reported in the paper?