fp8 GGUF version?
#2
by
jdc4429
- opened
Any chance of getting a GGUF fp8 version? fp16 is too large even for my 24GB GPU...
https://huggingface.co/Quazim0t0/ODB-14b-GGUF.q4_k_m
Only did a q4 quant this this one right now