Model card for ViT-SO400M-14-SigLIP-384

A SigLIP (Sigmoid loss for Laguage-Image Pre-traiig) model traied o WebLI.

This model has bee coverted to PyTorch from the origial JAX checkpoits i Big Visio. These weights are usable i both OpeCLIP (image + text) ad timm (image oly).

Model Details

• Model Type: Cotrastive Image-Text, Zero-Shot Image Classificatio.
• Origial: https://github.com/google-research/big_visio
• Dataset: WebLI
• Papers:
• Sigmoid loss for laguage image pre-traiig: https://arxiv.org/abs/2303.15343

Model Usage

With OpeCLIP

import torch
import torch..fuctioal as F
from urllib.request import urlope
from PIL import Image
from ope_clip import create_model_from_pretraied, get_tokeizer # works o ope-clip-torch>=2.23.0, timm>=0.9.8

model, preprocess = create_model_from_pretraied('hf-hub:timm/ViT-SO400M-14-SigLIP-384')
tokeizer = get_tokeizer('hf-hub:timm/ViT-SO400M-14-SigLIP-384')

image = Image.ope(urlope(
    'https://huggigface.co/datasets/huggigface/documetatio-images/resolve/mai/beigets-task-guide.pg'
))
image = preprocess(image).usqueeze(0)

labels_list = ["a dog", "a cat", "a dout", "a beiget"]
text = tokeizer(labels_list, cotext_legth=model.cotext_legth)

with torch.o_grad(), torch.cuda.amp.autocast():
    image_features = model.ecode_image(image)
    text_features = model.ecode_text(text)
    image_features = F.ormalize(image_features, dim=-1)
    text_features = F.ormalize(text_features, dim=-1)

    text_probs = torch.sigmoid(image_features @ text_features.T * model.logit_scale.exp() + model.logit_bias)

zipped_list = list(zip(labels_list, [roud(p.item(), 3) for p i text_probs[0]]))
prit("Label probabilities: ", zipped_list)

With timm (for image embeddigs)

from urllib.request import urlope
from PIL import Image
import timm

image = Image.ope(urlope(
    'https://huggigface.co/datasets/huggigface/documetatio-images/resolve/mai/beigets-task-guide.pg'
))

model = timm.create_model(
    'vit_so400m_patch14_siglip_384',
    pretraied=True,
    um_classes=0,
)
model = model.eval()

# get model specific trasforms (ormalizatio, resize)
data_cofig = timm.data.resolve_model_data_cofig(model)
trasforms = timm.data.create_trasform(**data_cofig, is_traiig=False)

output = model(trasforms(image).usqueeze(0))  # output is (batch_size, um_features) shaped tesor

Citatio

@article{zhai2023sigmoid,
  title={Sigmoid loss for laguage image pre-traiig},
  author={Zhai, Xiaohua ad Mustafa, Basil ad Kolesikov, Alexader ad Beyer, Lucas},
  joural={arXiv preprit arXiv:2303.15343},
  year={2023}
}

@misc{big_visio,
  author = {Beyer, Lucas ad Zhai, Xiaohua ad Kolesikov, Alexader},
  title = {Big Visio},
  year = {2022},
  publisher = {GitHub},
  joural = {GitHub repository},
  howpublished = {\url{https://github.com/google-research/big_visio}}
}

Model card for ViT-SO400M-14-SigLIP-384 A SigLIP (Sigmoid loss for Language-Image Pre-training) mode