Exl2 quants for Llama-Guard-3-1B
This is a quantized model from the llama guard series for use with exllamav2.
Please refer to the original model card for instructions on how to use this model.
Only use this version for testing, BF16 weights recommended for deployment or at least 8bpw.
BPW:
4.0
4.5
5.0
6.0
6.5
8.0
measurement.json
How to download:
oobabooga's downloader
use something like download-model.py to download with python requests.
Install requirements:
pip install requests tqdm
Example for downloading 8bpw:
python download-model.py Anthonyg5005/Llama-Guard-3-1B-exl2:8.0bpw
huggingface-cli
You may also use huggingface-cli
To install it, install python hf-hub
pip install huggingface-hub
Example for 8bpw:
huggingface-cli download Anthonyg5005/Llama-Guard-3-1B-exl2 --local-dir Llama-Guard-3-1B-exl2-exl2-8bpw --revision 8.0bpw
Git LFS (not recommended)
I would recommend the http downloaders over using git, they can resume downloads if failed and are much easier to work with.
Make sure to have git and git LFS installed.
Example for 8bpw download with git:
Have LFS file skip disabled
# windows
set GIT_LFS_SKIP_SMUDGE=0
# linux
export GIT_LFS_SKIP_SMUDGE=0
Clone repo branch
git clone https://huggingface.co/Anthonyg5005/Llama-Guard-3-1B-exl2 -b 8.0bpw
Inference Providers
NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API:
The HF Inference API does not support text-generation models for exllamav2 library.
Model tree for Anthonyg5005/Llama-Guard-3-1B-exl2
Base model
meta-llama/Llama-Guard-3-1B