README.md

metadata

base_model:
  - cognitivecomputations/Dolphin3.0-R1-Mistral-24B
datasets:
  - eaddario/imatrix-calibration
language:
  - en
license:
  - apache-2.0
pipeline_tag: text-generation
tags:
  - gguf
  - quant

GGUF and "i-matrix" quantized versions of cognitivecomputations/Dolphin3.0-R1-Mistral-24B

Using LLaMA C++ release b4722 for quantization.

Original model: cognitivecomputations/Dolphin3.0-R1-Mistral-24B

From the model creator:

Dolphin 3.0 R1 is the next generation of the Dolphin series of instruct-tuned models. Designed to be the ultimate general purpose local model, enabling coding, math, agentic, function calling, and general use cases. The R1 version has been trained for 3 epochs to reason using 800k reasoning traces from the Dolphin-R1 dataset.

Dolphin aims to be an uncensored general purpose reasoning instruct model, similar to the models behind ChatGPT, Claude, Gemini. But these models present problems for businesses seeking to include AI in their products.

• They maintain control of the system prompt, deprecating and changing things as they wish, often causing software to break.
• They maintain control of the model versions, sometimes changing things silently, or deprecating older models that your business relies on.
• They maintain control of the alignment, and in particular the alignment is one-size-fits all, not tailored to the application.
• They can see all your queries and they can potentially use that data in ways you wouldn't want. Dolphin, in contrast, is steerable and gives control to the system owner. You set the system prompt. You decide the alignment. You have control of your data. Dolphin does not impose its ethics or guidelines on you. You are the one who decides the guidelines.
• Dolphin belongs to YOU, it is your tool, an extension of your will. Just as you are personally responsible for what you do with a knife, gun, fire, car, or the internet, you are the creator and originator of any content you generate with Dolphin.

From Eric Hartford's, the creator of the Dolphin model series, Uncensored Models:

Most of these models (for example, Alpaca, Vicuna, WizardLM, MPT-7B-Chat, Wizard-Vicuna, GPT4-X-Vicuna) have some sort of embedded alignment. For general purposes, this is a good thing. This is what stops the model from doing bad things, like teaching you how to cook meth and make bombs. But what is the nature of this alignment? And, why is it so?

The reason these models are aligned is that they are trained with data that was generated by ChatGPT, which itself is aligned by an alignment team at OpenAI. As it is a black box, we don't know all the reasons for the decisions that were made, but we can observe it generally is aligned with American popular culture, and to obey American law, and with a liberal and progressive political bias.

All quantized versions were generated using an appropriate imatrix created from datasets available at eaddario/imatrix-calibration.

At its core, an Importance Matrix (imatrix) is a table or, more broadly, a structured representation that scores the relative importance of different features or parameters in a machine learning model. It essentially quantifies the "impact" each feature has on a specific outcome, prediction, or relationship being modeled.

The process to produce the quantized GGUF models is roughly as follows:

1. Convert the the original model's safetensors into GGUF F16*
2. Estimate the Perplexity score for the F16 model (base) using wikitext-2-raw-v1, and record the logits
3. Generate the imatrix for each calibration dataset
4. Create quantized versions of the base model using each imatrix per quant type
5. Calculate the Perplexity and KL Divergence scores for each quantized model (scores)
6. For each quant type, keep the version with the best (usually the lowest) scores

*BF16 would be preferred, but Apple's GPUs don't support it yet, and therefore any operations are executed in the CPU, making it unacceptably slow. This is expected to change in the near term but until then, if you are using Apple kit avoid using any models tagged BF16

Motivation

An area of ongoing personal research is to optimize the inference performance of LLMs when deployed in resource-constrained environments like, for example, commodity hardware, personal desktops/laptops, edge devices, etc.

The process of quantization reduces the precision of the model's weights, leading to significant reductions in model size, memory needs and computational requirements (a good thing), but this however comes at the expense of a loss in the model's capabilities and accuracy (a bad thing!).

By producing imatrix optimized quantized models, we can maintain inference efficiency whilst reducing memory size and CPU/GPU processing requirements. This optimization is crucial for deploying LLMs on devices with limited hardware capabilities, such as mobile phones or edge devices, without sacrificing significant accuracy.

Models

Filename	Quant type	Size	Perplexity (μ)	ln(PPL(Q)/PPL(base))	KL Divergence (μ)	Description
Dolphin3.0-R1-Mistral-24B-F16	F16	47.20G	23.352232 ±0.220841	N/A	N/A	16-bit standard IEEE 754 half-precision floating-point number
Dolphin3.0-R1-Mistral-24B-Q8_0	Q8_0	25.10G	23.519878 ±0.223215	99.96%	0.002318 ±0.000013	Extremely high quality, generally unneeded but max available quant
Dolphin3.0-R1-Mistral-24B-Q6_K	Q6_K	19.30G	24.415348 ±0.234385	99.79%	0.015503 ±0.000075	Very high quality, near perfect, recommended
Dolphin3.0-R1-Mistral-24B-Q5_K_M	Q5_K_M	16.80G	24.304562 ±0.233045	99.74%	0.019001 ±0.000084	High quality
Dolphin3.0-R1-Mistral-24B-Q5_K_S	Q5_K_S	16.30G				High quality, recommended
Dolphin3.0-R1-Mistral-24B-IQ4_NL	IQ4_NL	13.50G	23.491557 ±0.221754	99.65%	0.025519 ±0.000102	Good quality, new method (super-blocks with 256 weights), recommended
Dolphin3.0-R1-Mistral-24B-Q4_K_M	Q4_K_M	14.30G	24.940671 ±0.241913	99.60%	0.030244 ±0.000124	Good quality, default size for must use cases, recommended
Dolphin3.0-R1-Mistral-24B-Q4_K_S	Q4_K_S	13.50G	23.315818 ±0.220552	99.50%	0.037680 ±0.000153	Good quality, best choice in the Q4 series if RAM is scarce, recommended
Dolphin3.0-R1-Mistral-24B-IQ3_M	IQ3_M	9.90G	26.172864 ±0.256006	99.01%	0.078326 ±0.000305	Medium-low quality, new method with decent performance comparable to Q3_K_M
Dolphin3.0-R1-Mistral-24B-IQ3_S	IQ3_S	9.70G	26.143038 ±0.254443	98.95%	0.082486 ±0.000324	Lower quality, new method with decent performance, sligthly better than Q3_K_S
Dolphin3.0-R1-Mistral-24B-Q3_K_L	Q3_K_L	12.40G	24.550639 ±0.235362	99.30%	0.053089 ±0.000213	Good quality, good choice for low RAM availability, recommended
Dolphin3.0-R1-Mistral-24B-Q3_K_M	Q3_K_M	11.50G	24.806925 ±0.237610	99.16%	0.063828 ±0.000254	Medium-low quality
Dolphin3.0-R1-Mistral-24B-Q3_K_S	Q3_K_S	9.7G	39.396539 ±0.409082	94.50%	0.448856 ±0.001660	Lower quality but may be usable in certain cases

I find that quantizations below Q3/IQ3 are not fit for my purposes and therefore do not usually generate them, but happy to provide other quants on request.

Metrics used

Perplexity: one of the key metrics used in NLP evaluation. It measures the quality of a language model by evaluating how well it predicts the next token given a particular sequence of words. A PPL of 1 indicates an exact match between predicted and actual, whereas values greater than one indicate a degree of "surprise" the generated token differs from the expected.

Kullback–Leibler (KL) Divergence: a statistical measure of how much a probability distribution differs from another. When quantizing models (or altering the original tensors in any way for that matter), the closest we can preserve the weights' probability distribution to the orignal model the better, thus the closest to 0 the better.

Credits

A big Thank You! to Colin Kealty for the many contributions and for being one of the best sources of high quality quantized models available in Hugginface, and a really big Thank You! to Georgi Gerganov for his amazing work with llama.cpp and the gguf file format.