danielhanchen commited on
Commit
67bbdce
·
verified ·
1 Parent(s): 644e4cf

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +9 -9
README.md CHANGED
@@ -15,11 +15,11 @@ tags:
15
  Or you can view more detailed instructions here: [unsloth.ai/blog/deepseek-r1](https://unsloth.ai/blog/deepseek-r1)
16
  1. Do not forget about `<|User|>` and `<|Assistant|>` tokens! - Or use a chat template formatter
17
  2. Obtain the latest `llama.cpp` at https://github.com/ggerganov/llama.cpp
18
- 3. Example with Q8_0 K quantized cache **Notice -no-cnv disables auto conversation mode**
19
  ```bash
20
  ./llama.cpp/llama-cli \
21
- --model unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M.gguf \
22
- --cache-type-k q8_0 \
23
  --threads 16 \
24
  --prompt '<|User|>What is 1+1?<|Assistant|>' \
25
  -no-cnv
@@ -38,12 +38,12 @@ Or you can view more detailed instructions here: [unsloth.ai/blog/deepseek-r1](h
38
  4. If you have a GPU (RTX 4090 for example) with 24GB, you can offload multiple layers to the GPU for faster processing. If you have multiple GPUs, you can probably offload more layers.
39
  ```bash
40
  ./llama.cpp/llama-cli \
41
- --model unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M.gguf
42
- --cache-type-k q8_0
43
- --threads 16
44
- --prompt '<|User|>What is 1+1?<|Assistant|>'
45
- --n-gpu-layers 20 \
46
- -no-cnv
47
  ```
48
 
49
  # Finetune LLMs 2-5x faster with 70% less memory via Unsloth!
 
15
  Or you can view more detailed instructions here: [unsloth.ai/blog/deepseek-r1](https://unsloth.ai/blog/deepseek-r1)
16
  1. Do not forget about `<|User|>` and `<|Assistant|>` tokens! - Or use a chat template formatter
17
  2. Obtain the latest `llama.cpp` at https://github.com/ggerganov/llama.cpp
18
+ 3. Example with Q5_0 K quantized cache **Notice -no-cnv disables auto conversation mode**
19
  ```bash
20
  ./llama.cpp/llama-cli \
21
+ --model unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q2_K_XS.gguf \
22
+ --cache-type-k q5_0 \
23
  --threads 16 \
24
  --prompt '<|User|>What is 1+1?<|Assistant|>' \
25
  -no-cnv
 
38
  4. If you have a GPU (RTX 4090 for example) with 24GB, you can offload multiple layers to the GPU for faster processing. If you have multiple GPUs, you can probably offload more layers.
39
  ```bash
40
  ./llama.cpp/llama-cli \
41
+ --model unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q2_K_XS.gguf \
42
+ --cache-type-k q5_0 \
43
+ --threads 16
44
+ --prompt '<|User|>What is 1+1?<|Assistant|>'
45
+ --n-gpu-layers 20 \
46
+ -no-cnv
47
  ```
48
 
49
  # Finetune LLMs 2-5x faster with 70% less memory via Unsloth!