SigLIP and SigLIP 2

SiglipModel

transformers' SiglipModel, the image-text dual encoder of SigLIP (Zhai et al., 2023) and of the SigLIP 2 checkpoints of fixed resolution (google/siglip2-base-patch16-224), which siglip_model builds: a SiglipTextModel text_model, a SiglipVisionModel vision_model, and the logit_scale and logit_bias of their similarities.

(model::SiglipModel)(input_ids, pixel_values; attention_mask = nothing,
                     interpolate_pos_encoding = false)
(model::SiglipModel)(inputs::NamedTuple)

The similarities of the texts input_ids (seqlen, numtexts), 1-based token indices, and the images pixel_values (width, height, channels, num_images), as a NamedTuple with the keys of transformers' SiglipOutput, in Julia's layout (PyTorch's dimensions reversed):

• logits_per_image (numtexts, numimages): the logit that each text describes each image, exp(logit_scale) * cos + logit_bias of the cosine similarity of their embeddings, whose sigmoid is the probability (SigLIP's sigmoid loss scores each pair alone, so the probabilities of the texts of an image don't sum to 1);
• logits_per_text (numimages, numtexts), its transpose;
• text_embeds (projectionsize, numtexts) and image_embeds (hidden, numimages): the `pooleroutput`s of the text and vision models, normalized to unit length;
• text_model_output and vision_model_output: the outputs of the text and vision models, (; last_hidden_state, pooler_output).

The texts are padded on the right to the length of the model (64 tokens), as SigLIP was trained, and attend to the padding unless attention_mask excludes it; the processor of a checkpoint, a SiglipProcessor, pads them so and returns no mask, as transformers'. The model takes its output as it is (model(inputs)). get_text_features and get_image_features encode texts or images alone.

using Metalhead, Flux, HuggingFaceTokenizers
model = create_model("google/siglip-base-patch16-224")
proc = create_processor("google/siglip-base-patch16-224")
image = rand(UInt8, 640, 480, 3)                       # width × height × channels
inputs = proc(; images = image, text = ["a photo of a cat", "a photo of a dog"])
probs = sigmoid.(model(inputs).logits_per_image)       # 2 × 1

Siglip2Model

transformers' Siglip2Model, the image-text dual encoder of the NaFlex checkpoints of SigLIP 2 (Tschannen et al., 2025), such as google/siglip2-base-patch16-naflex, which siglip2_model builds: a SiglipTextModel text_model, a Siglip2VisionModel vision_model, which takes images of any aspect ratio as patches, and the logit_scale and logit_bias of their similarities. The SigLIP 2 checkpoints of fixed resolution (google/siglip2-base-patch16-224) are SiglipModels.

(model::Siglip2Model)(input_ids, pixel_values; spatial_shapes,
                      pixel_attention_mask = nothing, attention_mask = nothing)
(model::Siglip2Model)(inputs::NamedTuple)

The similarities of the texts input_ids (seqlen, numtexts), 1-based token indices, and the images pixel_values, spatial_shapes and pixel_attention_mask (see Siglip2VisionModel), as a NamedTuple with the keys of transformers' Siglip2Output (those of SiglipModel): logits_per_image (numtexts, numimages), whose sigmoid is the probability that each text describes each image, logits_per_text, its transpose, the normalized text_embeds and image_embeds, and the outputs of the two models, text_model_output and vision_model_output. The model takes the output of the processor of a checkpoint, a Siglip2Processor, as it is (model(inputs)).

using Metalhead, Flux, HuggingFaceTokenizers
model = create_model("google/siglip2-base-patch16-naflex")
proc = create_processor("google/siglip2-base-patch16-naflex")
image = rand(UInt8, 640, 480, 3)                       # width × height × channels
inputs = proc(; images = image, text = ["a photo of a cat", "a photo of a dog"])
probs = sigmoid.(model(inputs).logits_per_image)       # 2 × 1

SiglipTextModel

transformers' SiglipTextModel, the text encoder of SigLIP and SigLIP 2, which siglip_text_model builds.

(model::SiglipTextModel)(input_ids; attention_mask = nothing)

The last_hidden_state (hidden, seqlen, batch), after the final norm, of the token indices `inputids(seq_len, batch), 1-based, and thepooleroutput` (projectionsize, batch), the projection by head of the features of the last token, which is padding in texts padded to the length of the model, as SigLIP was trained. Every token attends to every other, but to the padding after the ones of attention_mask.

SiglipVisionModel

transformers' SiglipVisionModel, the vision encoder of SigLIP and of the SigLIP 2 checkpoints of fixed resolution, which siglip_vision_model builds.

(model::SiglipVisionModel)(pixel_values; interpolate_pos_encoding = false)

The last_hidden_state (hidden, numpatches, batch), after the final norm, of the images `pixelvalues(width, height, channels, batch), and thepooleroutput(hidden, batch), the features that the attention-poolingheadpools from them. The images have theimagesizeof the model, unlessinterpolateposencoding`, which resizes the position embeddings bicubically to the grid of patches of the images, as transformers does.

Siglip2VisionModel

transformers' Siglip2VisionModel, the vision encoder of the NaFlex checkpoints of SigLIP 2, which siglip2_vision_model builds.

(model::Siglip2VisionModel)(pixel_values; spatial_shapes, pixel_attention_mask = nothing)
(model::Siglip2VisionModel)(inputs::NamedTuple)

The last_hidden_state (hidden, numpatches, batch), after the final norm, and the `pooleroutput(hidden, batch) that the attention-poolingheadpools from it, of images given as patches, as the image processor of a checkpoint makes them (a [Siglip2ImageProcessor`](@ref Metalhead.Transformers.Siglip2ImageProcessor), whose output the model also takes as it is):

• pixel_values (numchannels * patchsize^2, numpatches, batch): the patches of each image, row-major over the grid of the image, padded to `numpatches, each flattened as a Julia array of size(numchannels, patchsize, patch_size)` (channels first, then width);
• spatial_shapes (2, batch): the numbers of patches (height, width) of each image, to which the position embeddings of a square grid are resized, bilinearly and antialiased, as in transformers;
• pixel_attention_mask (num_patches, batch): 1 for the patches of the image, 0 for the padding after them, which no patch attends to (all patches are valid without it).

SiglipProcessor(image_processor, tokenizer)

transformers' processor of SigLIP: a SiglipImageProcessor and the tokenizer of the repository, HuggingFaceTokenizers.jl's Tokenizer, which create_processor loads from a repository such as "google/siglip-base-patch16-224" or "google/siglip2-base-patch16-224". The tokenizer needs using HuggingFaceTokenizers before create_processor; without it, tokenizer is nothing, and the processor takes images but no text.

(processor::SiglipProcessor)(; images = nothing, text = nothing, max_length = 64)

The inputs of a SiglipModel from images (one image or a vector of them, see SiglipImageProcessor) and text, as a NamedTuple with the keys of transformers' SiglipProcessor, in Julia's layout (PyTorch's dimensions reversed), which the model takes as it is (model(inputs)):

text is a string or a vector of strings, which the tokenizer encodes with its special tokens (the end-of-text token after the text), padded on the right to max_length tokens with the padding token of the tokenizer, as SigLIP was trained: as transformers' SiglipProcessor pads them with padding = "max_length", which the model cards pass (transformers pads none by default). The 64 tokens of the default are the length of the text models of the checkpoints, but for google/siglip-so400m-patch14-224, whose text model takes 16 (max_length = 16). A longer text is an error, as transformers doesn't truncate the texts of SigLIP. There is no attention_mask, as the tokenizers of the checkpoints return none (their model_input_names): the model attends to the padding, as it was trained.

The tokenizer of the SigLIP checkpoints (SiglipTokenizer) lowercases the texts and drops their punctuation before encoding them; their tokenizer.json does the same, but keeps the characters /, < and >, so that a text with them gets other tokens than in transformers. The SigLIP 2 checkpoints of fixed resolution (Gemma's tokenizer) give the same tokens as transformers.

using Metalhead, Flux, HuggingFaceTokenizers
model = create_model("google/siglip-base-patch16-224")
proc = create_processor("google/siglip-base-patch16-224")
image = rand(UInt8, 640, 480, 3)                       # width × height × channels
inputs = proc(; images = image, text = ["a photo of a cat", "a photo of a dog"])
probs = sigmoid.(model(inputs).logits_per_image)       # 2 × 1

SiglipImageProcessor(; do_resize = true, size = (; height = 224, width = 224),
                     resample = 3, do_rescale = true, rescale_factor = 1 / 255,
                     do_normalize = true, image_mean = (0.5, 0.5, 0.5),
                     image_std = (0.5, 0.5, 0.5), do_convert_rgb = true,
                     do_center_crop = false, do_pad = false)

transformers' image processor of SigLIP, with the defaults of its Python class; the image processor of a SiglipProcessor, with the settings of a repository (the SigLIP 2 checkpoints of fixed resolution resize bilinearly, resample = 2). Called on one image or a vector of images, it returns a NamedTuple:

An image is an array of size (width, height, channels) (torch's (channels, height, width) reversed) of integers from 0 to 255 (an 8-bit image, as UInt8) or of reals from 0 to 1. Each image is

1. converted to RGB with do_convert_rgb: a grey image (a matrix, or one channel, or two with alpha) has its grey repeated, and the alpha of RGBA is dropped, as PIL does;
2. resized to size with do_resize, with PIL's filter resample, 2 (bilinear) or 3 (bicubic), antialiased as torchvision resizes it;
3. multiplied by rescale_factor with do_rescale, and normalized with image_mean and image_std with do_normalize. An image of reals from 0 to 1 is taken as an 8-bit image already rescaled: do_rescale leaves it as it is, and without it the values are multiplied by 255.

An 8-bit image gives the same pixel_values as transformers, and an image of reals the same to Float32 precision. The settings are checked as the processor is made, and also take the forms that transformers takes: resample as the name of PIL's filter (:bicubic); image_mean and image_std as one value for all channels. An integer size, which resizes the shortest edge in transformers' SigLIP (default_to_square = False), the center crop and the padding aren't ported: do_center_crop and do_pad are false (or nothing, Python's None).

Siglip2Processor(image_processor, tokenizer)

transformers' processor of the NaFlex checkpoints of SigLIP 2: a Siglip2ImageProcessor and the tokenizer of the repository, HuggingFaceTokenizers.jl's Tokenizer, which create_processor loads from a repository such as "google/siglip2-base-patch16-naflex". The tokenizer needs using HuggingFaceTokenizers before create_processor; without it, tokenizer is nothing, and the processor takes images but no text.

(processor::Siglip2Processor)(; images = nothing, text = nothing, max_length = 64)

The inputs of a Siglip2Model from images (one image or a vector of them, see Siglip2ImageProcessor) and text, as a NamedTuple with the keys of transformers' Siglip2Processor, in Julia's layout (PyTorch's dimensions reversed), which the model takes as it is (model(inputs)):

where D = 3 * patch_size^2 (768). text is a string or a vector of strings, lowercased (Julia's lowercase), as transformers' Siglip2Tokenizer lowercases it, and encoded by the tokenizer with its special tokens (the end-of-text token after the text). The texts are padded on the right to max_length tokens with the padding token of the tokenizer, and longer ones are cut to max_length tokens, of which the last is the end-of-text token, as transformers' Siglip2Processor pads and truncates them by default (padding = "max_length", truncation = True, max_length = 64). There is no attention_mask, as the tokenizers of the checkpoints return none (their model_input_names): the model attends to the padding, as it was trained.

using Metalhead, Flux, HuggingFaceTokenizers
model = create_model("google/siglip2-base-patch16-naflex")
proc = create_processor("google/siglip2-base-patch16-naflex")
image = rand(UInt8, 640, 480, 3)                       # width × height × channels
inputs = proc(; images = image, text = ["a photo of a cat", "a photo of a dog"])
inputs.spatial_shapes                                  # (14, 18): 14 rows of 18 patches
probs = sigmoid.(model(inputs).logits_per_image)       # 2 × 1

Siglip2ImageProcessor(; do_resize = true, resample = 2, do_rescale = true,
                      rescale_factor = 1 / 255, do_normalize = true,
                      image_mean = (0.5, 0.5, 0.5), image_std = (0.5, 0.5, 0.5),
                      do_convert_rgb = false, patch_size = 16, max_num_patches = 256)

transformers' image processor of the NaFlex checkpoints of SigLIP 2, with the defaults of its Python class; the image processor of a Siglip2Processor, with the settings of a repository such as "google/siglip2-base-patch16-naflex". It keeps the aspect ratio of each image, which it cuts into at most max_num_patches patches. Called on one image or a vector of images, it returns the inputs of the vision model of SigLIP 2 (Siglip2VisionModel), which takes them as they are, as a NamedTuple in Julia's layout (PyTorch's dimensions reversed):

where D = 3 * patch_size^2 (768 for SigLIP 2). The patches of an image are row-major over its grid, and each is flattened as an array of size (3, patch_size, patch_size) (channels first, then width), as transformers' convert_image_to_patches flattens them.

An image is an array of size (width, height, channels) (torch's (channels, height, width) reversed) of integers from 0 to 255 (an 8-bit image, as UInt8) or of reals from 0 to 1. Each image is

1. converted to RGB with do_convert_rgb (false by default, Python's None): a grey image (a matrix, or one channel, or two with alpha) has its grey repeated, and the alpha of RGBA is dropped, as PIL does;
2. resized with do_resize, with PIL's filter resample, 2 (bilinear) or 3 (bicubic), antialiased as torchvision resizes it, to the largest size in multiples of patch_size with at most max_num_patches patches whose sides scale the image by the same factor, rounded up (transformers' get_image_size_for_max_num_patches). Without do_resize, the sides of an image must be multiples of patch_size;
3. multiplied by rescale_factor with do_rescale, and normalized with image_mean and image_std with do_normalize. An image of reals from 0 to 1 is taken as an 8-bit image already rescaled: do_rescale leaves it as it is, and without it the values are multiplied by 255;
4. cut into patches of patch_size × patch_size pixels, padded to max_num_patches.

An 8-bit image gives the same pixel_values as transformers, and an image of reals the same to Float32 precision. The settings are checked as the processor is made, and also take the forms that transformers takes: resample as the name of PIL's filter (:bilinear); image_mean and image_std as one value for all channels.