google

/

siglip2-base-patch16-224

like 31

Follow

Google 8.63k

I have the same issue using the AutoModel and AutoProcessor classes (both from transformers-4.49.0 package)

You need to install the latest version of transformers, for now pip install git+https://github.com/huggingface/transformers

pip install transformers==4.49.0 does not work.

@menglan There is no pypi package compatible with siglip2 checkpoints. Please use the latest version installed directly from the github, as mentioned above

I had this same issue and was just wondering when siglip2 checkpoints will be available as a pypi release?

Edit: I was using this and using the section Using the model yourself. I installed the version from github but that gives another error. Have pasted it below:

Asking to pad to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no padding.
Traceback (most recent call last):
  File "/rsp/data/ML/sarthakExperiments/test/venv/lib64/python3.9/site-packages/transformers/tokenization_utils_base.py", line 777, in convert_to_tensors
    tensor = as_tensor(value)
  File "/rsp/data/ML/sarthakExperiments/test/venv/lib64/python3.9/site-packages/transformers/tokenization_utils_base.py", line 739, in as_tensor
    return torch.tensor(value)
ValueError: expected sequence of length 10 at dim 1 (got 9)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/rsp/data/ML/sarthakExperiments/test/test.py", line 16, in <module>
    inputs = processor(text = texts,
  File "/rsp/data/ML/sarthakExperiments/test/venv/lib64/python3.9/site-packages/transformers/models/siglip/processing_siglip.py", line 108, in __call__
    encoding = self.tokenizer(
  File "/rsp/data/ML/sarthakExperiments/test/venv/lib64/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2877, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/rsp/data/ML/sarthakExperiments/test/venv/lib64/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2965, in _call_one
    return self.batch_encode_plus(
  File "/rsp/data/ML/sarthakExperiments/test/venv/lib64/python3.9/site-packages/transformers/tokenization_utils_base.py", line 3167, in batch_encode_plus
    return self._batch_encode_plus(
  File "/rsp/data/ML/sarthakExperiments/test/venv/lib64/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 587, in _batch_encode_plus
    return BatchEncoding(sanitized_tokens, sanitized_encodings, tensor_type=return_tensors)
  File "/rsp/data/ML/sarthakExperiments/test/venv/lib64/python3.9/site-packages/transformers/tokenization_utils_base.py", line 241, in __init__
    self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=pre

Just change AutoProcessor to AutoImageProcessor.

@kurisu0306 Tried that and it worked but the inputs only embed the image and not the texts which throws another error
Code:

inputs = processor(text = texts,
                   images = image,
                   padding = "max_length", 
                   return_tensors = "pt").to('cuda')

Then I pass these inputs to the model as:

with torch.no_grad():
    with torch.autocast('cuda'):
        outputs = model(**inputs)

But it throws the error:

.......
958, in forward
    raise ValueError("You have to specify input_ids")
ValueError: You have to specify input_ids

On having a look at the output after passing through the processor, this is what inputs has:

(Pdb) inputs.keys()
dict_keys(['pixel_values'])
(Pdb) inputs['pixel_values'].shape
torch.Size([1, 3, 224, 224])

Now it makes sense that my input image got encoded but shouldn't the texts be encoded as well? Maybe that is what is throwing that error?

You need to install the latest version of transformers, for now pip install git+https://github.com/huggingface/transformers

Is there anyone successed in running siglip2???

I've tried this way to install transformers, but then I get another error:

based on the new transformers, I've tried update the timm into version timm-1.0.13，and another error comes:

I'm struggling into the bug, looking forward to some advice, thanks!

my code is as follows:

from transformers import pipeline
from PIL import Image
import torch
from transformers import AutoModel, AutoProcessor
from transformers.image_utils import load_image


#### get feature
import torch
from transformers import AutoModel, AutoProcessor
from transformers.image_utils import load_image

# load the model and processor
ckpt = "xxxxx/siglip2-large-patch16-256"
model = AutoModel.from_pretrained(ckpt, device_map="auto").eval()

processor = AutoProcessor.from_pretrained(ckpt, use_fast=True)


# load the image
image = Image.open("./dog.jpg")
candidate_labels = ["2 cats", "2 dogs"]

inputs = processor(texts=candidate_labels, images=image, padding="max_length", return_tensors="pt").to(device)


with torch.no_grad():
    outputs = model(**inputs)

print(type(outputs))
print(dir(outputs))
print(outputs)
# run infernece
with torch.no_grad():
    image_embeddings = model.get_image_features(**inputs)    

print(image_embeddings.shape)