IntLLaMA, a fast and light quantization solution reduces gpu-memory requirement and improve computational efficiency while simultaneously preserving model intelligence. Specifically, IntLLaMA facilitates a quantization-friendly distribution of hidden-states by utilizing Random Centralization to address the asymmetry and mitigate the impact of outliers. Meanwhile, Hessian-weighted Singular Value Decomposition(HSVD) is further proposed to compensate for the performance degradation caused by representing the model weights using low bit-width. Benefits from RandC and HSVD, IntLLaMA quantize the weight into 4 bit-width, hidden-state into 8 bit-width sperately and close to full-precision performance in perplexity and MMLU accuracy.
- 2023-07-13: Release the code for LoRA instruct fine-tuing, More information can be found in
- 2023-07-13: Release a 4w8f ChatGLMv2-6B, which archieve in C-eval and speedup . The more detail can be found in Table1 .
- 2023-07-12: Release the code for convert a full-precision model to quantized model
IntLLaMA was inspired by several open source projects. We are grateful for these excellent projects and list them as follows:
- GPTQ
- AWQ
- Alpaca-LoRA
- Standard-Alpaca
IntLLaMA is released under the Apache 2.0 license.