eaddario
/

DeepSeek-R1-Distill-Qwen-7B-GGUF

+---
+base_model:
+- deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
+datasets:
+- eaddario/imatrix-calibration
+language:
+- en
+license:
+- mit
+pipeline_tag: text-generation
+tags:
+- gguf
+- quant
+---
+# GGUF and "i-matrix" quantized versions of deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
+Using [LLaMA C++](https://github.com/ggerganov/llama.cpp) release [b4601](https://github.com/ggerganov/llama.cpp/releases/tag/b4601) for quantization.
+Original model: [deepseek-ai/DeepSeek-R1-Distill-Qwen-7B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B)
+All quantized versions were generated using an appropriate imatrix created from datasets available at [eaddario/imatrix-calibration](https://huggingface.co/datasets/eaddario/imatrix-calibration).
+At its core, an Importance Matrix (imatrix) is a table or, more broadly, a structured representation that socres the relative importance of different features or parameters in a machine learning model. It essentially quantifies the "impact" each feature has on a specific outcome, prediction, or relationship being modeled.
+The process to produce the quantized [GGUF](https://huggingface.co/docs/hub/en/gguf) models is roughly as follows:
+1. Convert the the original model's safetensors into GGUF F16*
+2. Estimate the Perplexity score for the F16 model (base) using [wikitext-2-raw-v1](https://huggingface.co/datasets/Salesforce/wikitext/tree/main/wikitext-2-raw-v1), and record the [logits](https://huggingface.co/eaddario/DeepSeek-R1-Distill-Qwen-14B-GGUF/tree/main/logits)
+3. Generate the [imatrix](https://huggingface.co/eaddario/DeepSeek-R1-Distill-Qwen-14B-GGUF/tree/main/imatrix) for each calibration dataset
+4. Create quantized versions of the base model using each imatrix per quant type
+5. Calculate the Perplexity and KL Divergence scores for each quantized model [(logs)](https://huggingface.co/eaddario/Watt-Tool-8B-GGUF/tree/main/scores)
+6. For each quant type, keep the version with the best (usually the lowest) scores
+*[BF16](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format) would be preferred, but Apple's GPUs don't support it yet, and therefore any operations are executed in the CPU, making it unacceptably slow. This is expected to change in the near term but until then, if you are using Apple kit avoid using any models tagged BF16
+# Motivation
+An area of ongoing personal research is to optimize the inference performance of LLMs when deployed in resource-constrained environments like, for example, commodity hardware, personal desktops/laptops, edge devices, etc.
+The process of [quantization](https://huggingface.co/docs/optimum/en/concept_guides/quantization) reduces the precision of the model's weights, leading to significant reductions in model size, memory needs and computational requirements (a good thing), but this however comes at the expense of a loss in the model's capabilities and accuracy (a bad thing!).
+By producing imatrix optimized quantized models, we can maintain inference efficiency whilst reducing memory size and CPU/GPU processing requirements. This optimization is crucial for deploying LLMs on devices with limited hardware capabilities, such as mobile phones or edge devices, without sacrificing significant accuracy.
+# Models
+| Filename                                                                        | Quant type | Size | Perplexity (μ)      | ln(PPL(Q)/PPL(base)) | KL Divergence (μ)  | Description                                                                    |
+|---------------------------------------------------------------------------------|------------|------|---------------------|----------------------|--------------------|--------------------------------------------------------------------------------|
+| [DeepSeek-R1-Distill-Qwen-7B-F16](./DeepSeek-R1-Distill-Qwen-7B-F16.gguf)       | F16        | 14G  | 24.931431 ±0.241228 | N/A                  | N/A                | 16-bit standard IEEE 754 half-precision floating-point number                  |
+| [DeepSeek-R1-Distill-Qwen-7B-Q8_0](./DeepSeek-R1-Distill-Qwen-7B-Q8_0.gguf)     | Q8_0       | 7.5G | 25.118649 ±0.245079 | 99.91%               | 0.000361 ±0.000002 | Extremely high quality, generally unneeded but max available quant             |
+| [DeepSeek-R1-Distill-Qwen-7B-Q6_K](./DeepSeek-R1-Distill-Qwen-7B-Q6_K.gguf)     | Q6_K       | 5.8G | 25.139045 ±0.245198 | 99.87%               | 0.003335 ±0.000014 | Very high quality, near perfect, *recommended*                                 |
+| [DeepSeek-R1-Distill-Qwen-7B-Q5_K_M](./DeepSeek-R1-Distill-Qwen-7B-Q5_K_M.gguf) | Q5_K_M     | 5.1G | 25.331908 ±0.247609 | 99.81%               | 0.008227 ±0.000037 | High quality                                                                   |
+| [DeepSeek-R1-Distill-Qwen-7B-Q5_K_S](./DeepSeek-R1-Distill-Qwen-7B-Q5_K_S.gguf) | Q5_K_S     | 4.9G | 25.403544 ±0.248578 | 99.79%               | 0.009320 ±0.000039 | High quality, *recommended*                                                    |
+| [DeepSeek-R1-Distill-Qwen-7B-IQ4_NL](./DeepSeek-R1-Distill-Qwen-7B-IQ4_NL.gguf) | IQ4_NL     | 4.1G | 25.645935 ±0.251542 | 99.48%               | 0.035420 ±0.000140 | Good quality, new method (super-blocks with 256 weights), *recommended*        |
+| [DeepSeek-R1-Distill-Qwen-7B-Q4_K_M](./DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf) | Q4_K_M     | 4.4G | 25.409863 ±0.248331 | 99.58%               | 0.027115 ±0.000104 | Good quality, default size for must use cases, *recommended*                   |
+| [DeepSeek-R1-Distill-Qwen-7B-Q4_K_S](./DeepSeek-R1-Distill-Qwen-7B-Q4_K_S.gguf) | Q4_K_S     | 4.2G | 25.379497 ±0.247973 | 99.52%               | 0.032033 ±0.000122 | Good quality, best choice in the Q4 series if RAM is scarce, *recommended*     |
+| [DeepSeek-R1-Distill-Qwen-7B-IQ3_M](./DeepSeek-R1-Distill-Qwen-7B-IQ3_M.gguf)   | IQ3_M      | 3.3G | 26.530303 ±0.256822 | 98.58%               | 0.110374 ±0.000408 | Medium-low quality, new method with decent performance comparable to Q3_K_M    |
+| [DeepSeek-R1-Distill-Qwen-7B-IQ3_S](./DeepSeek-R1-Distill-Qwen-7B-IQ3_S.gguf)   | IQ3_S      | 3.3G | 26.638550 ±0.259075 | 98.51%               | 0.115060 ±0.000427 | Lower quality, new method with decent performance, sligthly better than Q3_K_S |
+| [DeepSeek-R1-Distill-Qwen-7B-Q3_K_L](./DeepSeek-R1-Distill-Qwen-7B-Q3_K_L.gguf) | Q3_K_L     | 3.8G | 26.027373 ±0.255226 | 98.91%               | 0.080142 ±0.000319 | Lower quality but usable, good for low RAM availability                        |
+| [DeepSeek-R1-Distill-Qwen-7B-Q3_K_M](./DeepSeek-R1-Distill-Qwen-7B-Q3_K_M.gguf) | Q3_K_M     | 3.5G | 25.889366 ±0.253265 | 98.77%               | 0.091519 ±0.000361 | Medium-low quality                                                             |
+| [DeepSeek-R1-Distill-Qwen-7B-Q3_K_S](./DeepSeek-R1-Distill-Qwen-7B-Q3_K_S.gguf) | Q3_K_S     | 3.3G | 26.552814 ±0.258782 | 97.86%               | 0.164859 ±0.000610 | Lower quality but may be usable in certain cases                               |
+I find that quantizations below Q3/IQ3 are not fit for my purposes and therefore do not usually generate them, but happy to provide other quants on request.
+# Metrics used
+**[Perplexity](https://huggingface.co/docs/transformers/en/perplexity):** one of the key metrics used in NLP evaluation. It measures the quality of a language model by evaluating how well it predicts the next token given a particular sequence of words. A PPL of **1** indicates an exact match between predicted and actual, whereas values greater than one indicate a degree of "surprise" the generated token differs from the expected.
+**[Kullback–Leibler (KL) Divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence):** a statistical measure of how much a probability distribution differs from another. When quantizing models (or altering the original tensors in any way for that matter), the closest we can preserve the weights' probability distribution to the orignal model the better, thus the closest to **0** the better.
+## Credits
+A big **Thank You!** to [Colin Kealty](https://huggingface.co/bartowski) for the many contributions and for being one of the best sources of high quality quantized models available in Hugginface, and a really big ***Thank You!*** to [Georgi Gerganov](https://github.com/ggerganov) for his amazing work with [llama.cpp](https://github.com/ggerganov/llama.cpp) and the [gguf](https://huggingface.co/docs/hub/en/gguf) file format.