| Quantization made by Richard Erkhov. | |
| [Github](https://github.com/RichardErkhov) | |
| [Discord](https://discord.gg/pvy7H8DZMG) | |
| [Request more models](https://github.com/RichardErkhov/quant_request) | |
| Qwen2.5-7B-Instruct-abliterated-v3 - GGUF | |
| - Model creator: https://huggingface.co/huihui-ai/ | |
| - Original model: https://huggingface.co/huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3/ | |
| | Name | Quant method | Size | | |
| | ---- | ---- | ---- | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q2_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q2_K.gguf) | Q2_K | 2.81GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_S.gguf) | Q3_K_S | 3.25GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q3_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q3_K.gguf) | Q3_K | 3.55GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_M.gguf) | Q3_K_M | 3.55GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_L.gguf) | Q3_K_L | 3.81GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.IQ4_XS.gguf) | IQ4_XS | 3.96GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q4_0.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_0.gguf) | Q4_0 | 4.13GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.IQ4_NL.gguf) | IQ4_NL | 4.16GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_K_S.gguf) | Q4_K_S | 4.15GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q4_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_K.gguf) | Q4_K | 4.36GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_K_M.gguf) | Q4_K_M | 4.36GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q4_1.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_1.gguf) | Q4_1 | 4.54GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q5_0.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_0.gguf) | Q5_0 | 4.95GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_K_S.gguf) | Q5_K_S | 4.95GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q5_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_K.gguf) | Q5_K | 5.07GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_K_M.gguf) | Q5_K_M | 5.07GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q5_1.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_1.gguf) | Q5_1 | 5.36GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q6_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q6_K.gguf) | Q6_K | 5.82GB | | |
| | [Qwen2.5-7B-Instruct-abliterated-v3.Q8_0.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q8_0.gguf) | Q8_0 | 7.54GB | | |
| Original model description: | |
| --- | |
| library_name: transformers | |
| license: apache-2.0 | |
| license_link: https://huggingface.co/huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3/blob/main/LICENSE | |
| language: | |
| - en | |
| pipeline_tag: text-generation | |
| base_model: Qwen/Qwen2.5-7B-Instruct | |
| tags: | |
| - chat | |
| - abliterated | |
| - uncensored | |
| --- | |
| # huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3 | |
| This is an uncensored version of [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) created with abliteration (see [remove-refusals-with-transformers](https://github.com/Sumandora/remove-refusals-with-transformers) to know more about it). | |
| This is a crude, proof-of-concept implementation to remove refusals from an LLM model without using TransformerLens. | |
| The test results are not very good, but compared to before, there is much less [garbled text](https://huggingface.co/huihui-ai/Qwen2.5-7B-Instruct-abliterated-v2/discussions/2). | |
| ## Usage | |
| You can use this model in your applications by loading it with Hugging Face's `transformers` library: | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| # Load the model and tokenizer | |
| model_name = "huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3" | |
| model = AutoModelForCausalLM.from_pretrained( | |
| model_name, | |
| torch_dtype="auto", | |
| device_map="auto" | |
| ) | |
| tokenizer = AutoTokenizer.from_pretrained(model_name) | |
| # Initialize conversation context | |
| initial_messages = [ | |
| {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."} | |
| ] | |
| messages = initial_messages.copy() # Copy the initial conversation context | |
| # Enter conversation loop | |
| while True: | |
| # Get user input | |
| user_input = input("User: ").strip() # Strip leading and trailing spaces | |
| # If the user types '/exit', end the conversation | |
| if user_input.lower() == "/exit": | |
| print("Exiting chat.") | |
| break | |
| # If the user types '/clean', reset the conversation context | |
| if user_input.lower() == "/clean": | |
| messages = initial_messages.copy() # Reset conversation context | |
| print("Chat history cleared. Starting a new conversation.") | |
| continue | |
| # If input is empty, prompt the user and continue | |
| if not user_input: | |
| print("Input cannot be empty. Please enter something.") | |
| continue | |
| # Add user input to the conversation | |
| messages.append({"role": "user", "content": user_input}) | |
| # Build the chat template | |
| text = tokenizer.apply_chat_template( | |
| messages, | |
| tokenize=False, | |
| add_generation_prompt=True | |
| ) | |
| # Tokenize input and prepare it for the model | |
| model_inputs = tokenizer([text], return_tensors="pt").to(model.device) | |
| # Generate a response from the model | |
| generated_ids = model.generate( | |
| **model_inputs, | |
| max_new_tokens=8192 | |
| ) | |
| # Extract model output, removing special tokens | |
| generated_ids = [ | |
| output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids) | |
| ] | |
| response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] | |
| # Add the model's response to the conversation | |
| messages.append({"role": "assistant", "content": response}) | |
| # Print the model's response | |
| print(f"Qwen: {response}") | |
| ``` | |
| ## Evaluations | |
| The following data has been re-evaluated and calculated as the average for each test. | |
| | Benchmark | Qwen2.5-7B-Instruct | Qwen2.5-7B-Instruct-abliterated-v3 | Qwen2.5-7B-Instruct-abliterated-v2 | Qwen2.5-7B-Instruct-abliterated | | |
| |-------------|---------------------|------------------------------------|------------------------------------|---------------------------------| | |
| | IF_Eval | 76.44 | 72.64 | **77.82** | 76.49 | | |
| | MMLU Pro | **43.12** | 39.14 | 42.03 | 41.71 | | |
| | TruthfulQA | 62.46 | 57.27 | 57.81 | **64.92** | | |
| | BBH | **53.92** | 50.67 | 53.01 | 52.77 | | |
| | GPQA | 31.91 | 31.65 | **32.17** | 31.97 | | |
| The script used for evaluation can be found inside this repository under /eval.sh, or click [here](https://huggingface.co/huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3/blob/main/eval.sh) | |