Hugging Face's logo

RichardErkhov
/
huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf

GGUF
Inference Endpoints
conversational
Model card Files Community
1
huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf
File size: 9,104 Bytes 
2004d94
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
 
Quantization made by Richard Erkhov.

[Github](https://github.com/RichardErkhov)

[Discord](https://discord.gg/pvy7H8DZMG)

[Request more models](https://github.com/RichardErkhov/quant_request)


Qwen2.5-7B-Instruct-abliterated-v3 - GGUF
- Model creator: https://huggingface.co/huihui-ai/
- Original model: https://huggingface.co/huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3/


| Name | Quant method | Size |
| ---- | ---- | ---- |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q2_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q2_K.gguf) | Q2_K | 2.81GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_S.gguf) | Q3_K_S | 3.25GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q3_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q3_K.gguf) | Q3_K | 3.55GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_M.gguf) | Q3_K_M | 3.55GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_L.gguf) | Q3_K_L | 3.81GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.IQ4_XS.gguf) | IQ4_XS | 3.96GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q4_0.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_0.gguf) | Q4_0 | 4.13GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.IQ4_NL.gguf) | IQ4_NL | 4.16GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_K_S.gguf) | Q4_K_S | 4.15GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q4_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_K.gguf) | Q4_K | 4.36GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_K_M.gguf) | Q4_K_M | 4.36GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q4_1.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_1.gguf) | Q4_1 | 4.54GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q5_0.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_0.gguf) | Q5_0 | 4.95GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_K_S.gguf) | Q5_K_S | 4.95GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q5_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_K.gguf) | Q5_K | 5.07GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_K_M.gguf) | Q5_K_M | 5.07GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q5_1.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_1.gguf) | Q5_1 | 5.36GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q6_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q6_K.gguf) | Q6_K | 5.82GB |
| [Qwen2.5-7B-Instruct-abliterated-v3.Q8_0.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q8_0.gguf) | Q8_0 | 7.54GB |




Original model description:
---
library_name: transformers
license: apache-2.0
license_link: https://huggingface.co/huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3/blob/main/LICENSE
language:
- en
pipeline_tag: text-generation
base_model: Qwen/Qwen2.5-7B-Instruct
tags:
- chat
- abliterated
- uncensored
---

# huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3


This is an uncensored version of [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) created with abliteration (see [remove-refusals-with-transformers](https://github.com/Sumandora/remove-refusals-with-transformers) to know more about it).
This is a crude, proof-of-concept implementation to remove refusals from an LLM model without using TransformerLens. 
The test results are not very good, but compared to before, there is much less [garbled text](https://huggingface.co/huihui-ai/Qwen2.5-7B-Instruct-abliterated-v2/discussions/2).

## Usage
You can use this model in your applications by loading it with Hugging Face's `transformers` library:


```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model and tokenizer
model_name = "huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Initialize conversation context
initial_messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}
]
messages = initial_messages.copy()  # Copy the initial conversation context

# Enter conversation loop
while True:
    # Get user input
    user_input = input("User: ").strip()  # Strip leading and trailing spaces

    # If the user types '/exit', end the conversation
    if user_input.lower() == "/exit":
        print("Exiting chat.")
        break

    # If the user types '/clean', reset the conversation context
    if user_input.lower() == "/clean":
        messages = initial_messages.copy()  # Reset conversation context
        print("Chat history cleared. Starting a new conversation.")
        continue

    # If input is empty, prompt the user and continue
    if not user_input:
        print("Input cannot be empty. Please enter something.")
        continue

    # Add user input to the conversation
    messages.append({"role": "user", "content": user_input})

    # Build the chat template
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )

    # Tokenize input and prepare it for the model
    model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

    # Generate a response from the model
    generated_ids = model.generate(
        **model_inputs,
        max_new_tokens=8192
    )

    # Extract model output, removing special tokens
    generated_ids = [
        output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
    ]
    response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

    # Add the model's response to the conversation
    messages.append({"role": "assistant", "content": response})

    # Print the model's response
    print(f"Qwen: {response}")

```

## Evaluations
The following data has been re-evaluated and calculated as the average for each test.

| Benchmark   | Qwen2.5-7B-Instruct | Qwen2.5-7B-Instruct-abliterated-v3 | Qwen2.5-7B-Instruct-abliterated-v2 | Qwen2.5-7B-Instruct-abliterated |
|-------------|---------------------|------------------------------------|------------------------------------|---------------------------------|
| IF_Eval     | 76.44               | 72.64                              | **77.82**                          | 76.49                           |
| MMLU Pro    | **43.12**           | 39.14                              | 42.03                              | 41.71                           |
| TruthfulQA  | 62.46               | 57.27                              | 57.81                              | **64.92**                       |
| BBH         | **53.92**           | 50.67                              | 53.01                              | 52.77                           |
| GPQA        | 31.91               | 31.65                              | **32.17**                          | 31.97                           |

The script used for evaluation can be found inside this repository under /eval.sh, or click [here](https://huggingface.co/huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3/blob/main/eval.sh)