HQQ

Half-Quadratic Quantization (HQQ) supports fast on-the-fly quantization for 8, 4, 3, 2, and even 1-bits. It doesn’t require calibration data, and it is compatible with any model modality (LLMs, vision, etc.).

HQQ further supports fine-tuning with PEFT and is fully compatible with torch.compile for even faster inference and training.

Install HQQ with the following command to get the latest version and to build its corresponding CUDA kernels.

pip install hqq

You can choose to either replace all the linear layers in a model with the same quantization config or dedicate a specific quantization config for specific linear layers.

from transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig

quant_config = HqqConfig(nbits=8, group_size=64)
model = transformers.AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B", 
    torch_dtype=torch.float16, 
    device_map="cuda", 
    quantization_config=quant_config
)

Backends

HQQ supports various backends, including pure PyTorch and custom dequantization CUDA kernels. These backends are suitable for older GPUs and PEFT/QLoRA training.

from hqq.core.quantize import *

HQQLinear.set_backend(HQQBackend.PYTORCH)

For faster inference, HQQ supports 4-bit fused kernels (torchao and Marlin) after a model is quantized. These can reach up to 200 tokens/sec on a single 4090. The example below demonstrates enabling the torchao_int4 backend.

from hqq.utils.patching import prepare_for_inference

prepare_for_inference("model", backend="torchao_int4")

Refer to the Backend guide for more details.

Resources

Read the Half-Quadratic Quantization of Large Machine Learning Models blog post for more details about HQQ.