Question about loading Pretrained AIMv2 ckpt #1
Description
Hi, thanks for your great work and effort! I'm trying to finetune the pretrained AIMv2 encoder for specific domain. However, I find that the keys of the pretrained weight provided on huggingface are not matched with the definition in the model part of this repository and the official AIMv2 repository.
The keys of the pretrained weight provided on huggingface are like 'encoder.layers.9.attention.k_proj.weight', while the layers in the model are like 'trunk.blocks.9.attn.qkv.weight'.
Could you kindly provide a script or some suggestions to transfer the pretrained weights? Many thanks!