cannot load int4

by babyta - opened Apr 19, 2024

Apr 19, 2024

ValueError: .to is not supported for 4-bit or 8-bit bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct dtype.

nielsr

Llava Hugging Face org Apr 22, 2024

Hi,

You need to pass device_map=auto to the from_pretrained method in case you pass a BitsAndBytesConfig.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment