eaddario/Hammer2.1-7b-GGUF

GGUF and "i-matrix" quantized versions of MadeAgents/Hammer2.1-7b

Using LLaMA C++ release b4601 for quantization.

Original model: MadeAgents/Hammer2.1-7b

From the model creators:

Hammer refers to a series of lightweight Large Action Models. Currently, we are releasing Hammer 2.1 models (0.5B, 1.5B, 3B, and 7B) with strong function calling capability. These models are based on the Qwen 2.5 coder series and utilize function masking techniques and other advanced technologies. Hammer 2.1 series bring significant enhancements, while still maintaining the basic functionality of Hammer 2.0's Single-Turn interaction and further strengthening other capabilities.

All quantized versions were generated using an appropriate imatrix created from datasets available at eaddario/imatrix-calibration.

At its core, an Importance Matrix (imatrix) is a table or, more broadly, a structured representation that scores the relative importance of different features or parameters in a machine learning model. It essentially quantifies the "impact" each feature has on a specific outcome, prediction, or relationship being modeled.

The process to produce the quantized GGUF models is roughly as follows:

1. Convert the the original model's safetensors into GGUF F16*
2. Estimate the Perplexity score for the F16 model (base) using wikitext-2-raw-v1, and record the logits
3. Generate the imatrix for each calibration dataset
4. Create quantized versions of the base model using each imatrix per quant type
5. Calculate the Perplexity and KL Divergence scores for each quantized model (scores)
6. For each quant type, keep the version with the best (usually the lowest) scores

*BF16 would be preferred, but Apple's GPUs don't support it yet, and therefore any operations are executed in the CPU, making it unacceptably slow. This is expected to change in the near term but until then, if you are using Apple kit avoid using any models tagged BF16

Motivation

An area of ongoing personal research is to optimize the inference performance of LLMs when deployed in resource-constrained environments like, for example, commodity hardware, personal desktops/laptops, edge devices, etc.

The process of quantization reduces the precision of the model's weights, leading to significant reductions in model size, memory needs and computational requirements (a good thing), but this however comes at the expense of a loss in the model's capabilities and accuracy (a bad thing!).

By producing imatrix optimized quantized models, we can maintain inference efficiency whilst reducing memory size and CPU/GPU processing requirements. This optimization is crucial for deploying LLMs on devices with limited hardware capabilities, such as mobile phones or edge devices, without sacrificing significant accuracy.

Models

Filename	Quant type	Size	Perplexity (μ)	ln(PPL(Q)/PPL(base))	KL Divergence (μ)	Description
Hammer2.1-7b-F16	F16	14G	24.931431 ±0.241228	N/A	N/A	16-bit standard IEEE 754 half-precision floating-point number
Hammer2.1-7b-Q8_0	Q8_0	7.5G	10.039829 ±0.072796	99.99%	0.000162 ±0.000001	Extremely high quality, generally unneeded but max available quant
Hammer2.1-7b-Q6_K	Q6_K	5.8G	10.063662 ±0.073025	99.97%	0.001360 ±0.000005	Very high quality, near perfect, recommended
Hammer2.1-7b-Q5_K_M	Q5_K_M	5.1G	10.071679 ±0.073178	99.93%	0.003501 ±0.000018	High quality
Hammer2.1-7b-Q5_K_S	Q5_K_S	4.9G	10.092163 ±0.073418	99.92%	0.004008 ±0.000021	High quality, recommended
Hammer2.1-7b-IQ4_NL	IQ4_NL	4.1G	10.188365 ±0.074008	99.71%	0.016340 ±0.000073	Good quality, new method (super-blocks with 256 weights), recommended
Hammer2.1-7b-Q4_K_M	Q4_K_M	4.4G	10.130249 ±0.073552	99.79%	0.012012 ±0.000059	Good quality, default size for must use cases, recommended
Hammer2.1-7b-Q4_K_S	Q4_K_S	4.1G	10.139939 ±0.073658	99.75%	0.014417 ±0.000070	Good quality, best choice in the Q4 series if RAM is scarce, recommended
Hammer2.1-7b-IQ3_M	IQ3_M	3.3G	10.349753 ±0.074664	99.18%	0.048768 ±0.000209	Medium-low quality, new method with decent performance comparable to Q3_K_M
Hammer2.1-7b-IQ3_S	IQ3_S	3.3G	10.401451 ±0.075250	99.14%	0.050746 ±0.000215	Lower quality, new method with decent performance, sligthly better than Q3_K_S
Hammer2.1-7b-Q3_K_L	Q3_K_L	3.8G	10.455858 ±0.076831	99.30%	0.039060 ±0.000178	Lower quality but usable, good for low RAM availability
Hammer2.1-7b-Q3_K_M	Q3_K_M	3.5G	10.469704 ±0.076871	99.20%	0.044002 ±0.000201	Medium-low quality
Hammer2.1-7b-Q3_K_S	Q3_K_S	3.3G	10.774104 ±0.078933	98.73%	0.072166 ±0.000301	Lower quality but may be usable in certain cases

I find that quantizations below Q3/IQ3 are not fit for my purposes and therefore do not usually generate them, but happy to provide other quants on request.

Metrics used

Perplexity: one of the key metrics used in NLP evaluation. It measures the quality of a language model by evaluating how well it predicts the next token given a particular sequence of words. A PPL of 1 indicates an exact match between predicted and actual, whereas values greater than one indicate a degree of "surprise" the generated token differs from the expected.

Kullback–Leibler (KL) Divergence: a statistical measure of how much a probability distribution differs from another. When quantizing models (or altering the original tensors in any way for that matter), the closest we can preserve the weights' probability distribution to the orignal model the better, thus the closest to 0 the better.

Credits

A big Thank You! to Colin Kealty for the many contributions and for being one of the best sources of high quality quantized models available in Hugginface, and a really big Thank You! to Georgi Gerganov for his amazing work with llama.cpp and the gguf file format.