Colflor

[ManuelFay]

From:
https://github.com/AhmedMasryKU/colflor

[ManuelFay]

Changed from the OG repo:

• casting the dtype to make loading in bfloat16 possible (pixel_values + full attention mask)
• remove use_cache was put in the processor init
• remove useless mock image
• linting
• WIP: train models with it to verify compatibility

todo:

• clarify the full attention mask shenanigan
• make the conversion code from the OG florence checkpoint available
• clarify why default torch weight init in the projection layer is not okay ?
• clarify why use_cache was put in the processor init
• clarify why the fast tokenizer does not work ?

[AhmedMasryKU]

Very excited to see ColFlor being merged into the main repo. I wanna add a few comments and clarifications.

The attention mask of the backbone model, Florence-2, only covers the text embeddings. That's why we can not directly apply it on the final output embeddings (image+text). Instead, I added a new parameter "full attention mask" to cover both the image and text. I think we can refactor this part in a better way.

Regarding the initialization of the linear projection layer, we can use the default torch initialization and remove the extra code I added. I added this custom initialization when I was debugging the convergence issues, but forgot to remove it.

I also used the default tokenizer and processor from the original Florence-2-base checkpoint. (https://huggingface.co/microsoft/Florence-2-base).

[ManuelFay]

Probably mergeable when we clean up the weird "attention mask things" and we run trainings

[tonywu71]

1. One important remark: shouldn't we rename the model from colflor to colflor2? This will:

• make it ISO with ColQwen2
• make it easier if we ever want to use our ColVision architecture for an hypothetical Florence3 model → ColFlor3.

Personlly, I think it's worth doing the proper renaming. What do you think?

2. Can you update the CHANGELOG too please?

Otherwise, LGTM! Thanks! 👌🏼

[tonywu71]

Looks this code is adapted microsoft/Florence-2-large from the Hf Hub. If yes, I would add the original link on top of the file. Moreover, I'm not a big fan of # ruff: noqa so I think we could get rid of it.

You might need to apply the ruff linter after this change!

Suggested change

	# ruff: noqa
	# coding=utf-8
	# Copyright 2024 Microsoft and the HuggingFace Inc. team. All rights reserved.
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software
	# distributed under the License is distributed on an "AS IS" BASIS,
	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	# See the License for the specific language governing permissions and
	# limitations under the License.
	# This code was copied and adapted from the "microsoft/Florence-2-large" model: https://huggingface.co/microsoft/Florence-2-large/blob/main/configuration_florence2.py
	# coding=utf-8
	# Copyright 2024 Microsoft and the HuggingFace Inc. team. All rights reserved.
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software
	# distributed under the License is distributed on an "AS IS" BASIS,
	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	# See the License for the specific language governing permissions and
	# limitations under the License.

[tonywu71]

Suggested change

	import warnings
	""" Florence-2 configuration"""
	"""Florence-2 configuration"""
	import warnings

[tonywu71]

Is it too late to allow the user to change the output dimension?

[tonywu71]

Same as before. Might need to run ruff too!

Suggested change

	# ruff: noqa
	# This code was copied and adapted from the "microsoft/Florence-2-large" model: https://huggingface.co/microsoft/Florence-2-large/blob/main/modeling_florence2.py
	# NOTE: Disable ruff to keep the original file style.
	# ruff: noqa

[tonywu71]

Same as before.

Suggested change

	# ruff: noqa
	# This code was copied and adapted from the "microsoft/Florence-2-large" model: https://huggingface.co/microsoft/Florence-2-large/blob/main/processing_florence2.py
	# NOTE: Disable ruff to keep the original file style.
	# ruff: noqa

[tonywu71]

Can we make the two imports absolute? Not super important, but i'd be great to be ISO with the rest of the codebase.

Suggested change

	from colpali_engine.utils.processing_utils import BaseVisualRetrieverProcessor
	from .processing_florence2 import Florence2Processor
	from colpali_engine.models.florence2.colflor.processing_florence2 import Florence2Processor
	from colpali_engine.utils.processing_utils import BaseVisualRetrieverProcessor

[tonywu71]

Suggested change

	Processor for ColPali.
	Processor for ColFlor.