Does any MLLM support neat packing?

[xiaosu-zhu]

Reminder

• I have read the README and searched the existing issues.

System Info

llamafactory version: 0.9.2.dev0
Platform: Linux-5.4.0-202-generic-x86_64-with-glibc2.31
Python version: 3.10.15
PyTorch version: 2.1.0+cu121 (GPU)
Transformers version: 4.46.1
Datasets version: 3.1.0
Accelerate version: 1.0.1
PEFT version: 0.12.0
TRL version: 0.9.6
GPU type: NVIDIA A800-SXM4-80GB
DeepSpeed version: 0.15.4

Reproduction

I have noticed that, if neat_packing is turned on, the script will check whether the model is supported in model/model_utils/packing.py:

def configure_packing(
    config: "PretrainedConfig", model_args: "ModelArguments", is_trainable: bool
) -> None:
    if not is_trainable or not model_args.block_diag_attn:
        return

    model_type = getattr(config, "model_type", None)
    ############# HERE ################
    if model_type in SUPPORTED_CLASS_FOR_BLOCK_DIAG_ATTN:
        _patch_for_block_diag_attn(model_type)
        logger.info_rank0(
            "Using block diagonal attention for sequence packing without cross-attention."
        )
    else:
        raise ValueError("Current model does not support block diagonal attention.")

Where the SUPPORTED_CLASS_FOR_BLOCK_DIAG_ATTN contains only LLMs but no MLLMs.

SUPPORTED_CLASS_FOR_BLOCK_DIAG_ATTN = {
    "cohere",
    "falcon",
    "gemma",
    "gemma2",
    "llama",
    "mistral",
    "phi",
    "phi3",
    "qwen2",
    "starcoder2",
}

I wonder if the neat_packing has already supported the multi-modal inputs or not? Is it possible to add the support?

Expected behavior

No response

Others

No response