Integrate marlin fp16/bf16-int4/int8 matrix multiplication kernel

[dacorvo]

Since the introduction of mixed-precision fp16-int4 MARLIN (Mixed Auto-Regressive Linear) kernels by IST-DASLab, new mixed-precision MARLIN kernels have been introduced for other data types.

In particular, mixed-precision fp16/bf16-int4/int8 kernels have been contributed to TGI and could be integrated in optimum-quanto as well with companion Int8MarlinQBytesTensor and Int4MarlinQBitsTensor to pack the weights.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[dacorvo]

The kernel has been integrated in quanto CUDA extension in https://github.com/huggingface/optimum-quanto/tree/add_marlin_int4_kernel (thanks to an initial work by @shcho1118).
It needs now to be fully integrated at inference.

[shovan777]

@dacorvo what should be done to integrate this at inference?

[dacorvo]

What is missing is a MarlinWeightsQBitsTensor class in the same spirit as AWQBitsTensor class, and a modification to QBitsTensor.create to select that class instead of AWQBitsTensor (because this kernel is more stable).
This is a bit involved as you have to do things right to stay compatible with serialization, gradients and compilation (all things done already in AWQBitsTensor).
Also, this requires writing some tests.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[dacorvo]

Done in #333