CLIP

CLIPModel

transformers' CLIPModel, the dual encoder of CLIP, built by clip_model: a CLIPTextModel and a CLIPVisionModel, whose pooled features text_projection and visual_projection project to a space they share, and the logit_scale of their similarities.

(model::CLIPModel)(input_ids, pixel_values; attention_mask = nothing,
                   interpolate_pos_encoding = false)
(model::CLIPModel)(inputs::NamedTuple)

The similarities of the texts input_ids (seqlen, numtexts), token indices, 1-based, padded on the right after the ones of attention_mask, and of the images pixel_values (width, height, channels, num_images), as the model's processor, a CLIPProcessor, makes them; or of the output of the processor, model(proc(; text, images)). Returns a NamedTuple with the keys of transformers' CLIPOutput, in Julia's layout (PyTorch's dimensions reversed):

The outputs of the encoders, unnormalised, are those of get_text_features and get_image_features.

CLIPTextModel

transformers' CLIPTextModel, the text encoder of CLIP, built by clip_text_model.

(model::CLIPTextModel)(input_ids; attention_mask = nothing)

The last_hidden_state (hiddensize, seqlen, batchsize), after the final norm, and the `pooleroutput(hidden_size, batch_size), its features at the end-of-text token of each sequence, of the token indicesinputids` (seqlen, batchsize), 1-based, padded on the right after the ones of `attentionmask`. The attention is causal.

CLIPVisionModel

transformers' CLIPVisionModel, the vision encoder of CLIP, a ViT, built by clip_vision_model.

(model::CLIPVisionModel)(pixel_values; interpolate_pos_encoding = false)

The last_hidden_state (hiddensize, 1 + numpatches, batchsize), the class token then the patches, row by row, and the `pooleroutput(hidden_size, batch_size), the class token after the final norm, of the imagespixelvalues` (width, height, channels, batchsize), as the model's processor makes them. The images must have the image_size of the model, unless interpolate_pos_encoding = true, which resizes the position embeddings to the grid of their patches, bicubically.

CLIPTextModelWithProjection

transformers' CLIPTextModelWithProjection, the text encoder of CLIP with its projection, built by clip_text_model_with_projection.

(model::CLIPTextModelWithProjection)(input_ids; attention_mask = nothing)

The text_embeds (projectiondim, batchsize), the projected pooler_output, and the last_hidden_state of its CLIPTextModel.

CLIPVisionModelWithProjection

transformers' CLIPVisionModelWithProjection, the vision encoder of CLIP with its projection, built by clip_vision_model_with_projection.

(model::CLIPVisionModelWithProjection)(pixel_values; interpolate_pos_encoding = false)

The image_embeds (projectiondim, batchsize), the projected pooler_output, and the last_hidden_state of its CLIPVisionModel.

clip_model(; text_config = nothing, vision_config = nothing, projection_dim = 512,
           logit_scale_init_value = 2.6592, initializer_factor = 1.0,
           text_config_dict = nothing, vision_config_dict = nothing)

Port of transformers' CLIPModel, the dual encoder of CLIP (Radford et al., 2021), built from its CLIPConfig: the text encoder clip_text_model of text_config and the vision encoder clip_vision_model of vision_config (NamedTuples of the fields of their configs, or nothing for transformers' defaults, a ViT-B/32), whose pooled features are projected to projection_dim. As in transformers, text_config_dict and vision_config_dict, which older config.json files have, replace text_config and vision_config: their fields over the defaults of the config classes.

See CLIPModel for its inputs and outputs, and CLIPProcessor, which create_processor loads, for its inputs.

clip_text_model(; vocab_size = 49408, hidden_size = 512, intermediate_size = 2048,
    projection_dim = 512, num_hidden_layers = 12, num_attention_heads = 8,
    max_position_embeddings = 77, hidden_act = "quick_gelu", layer_norm_eps = 1e-5,
    attention_dropout = 0.0, initializer_range = 0.02, initializer_factor = 1.0,
    eos_token_id = 49407)

Port of transformers' CLIPTextModel, the text encoder of CLIP, built from the fields of a CLIPTextConfig (projection_dim is unused): a causal transformer over the token indices (seq_len, batch_size) (1-based), pooled at the first end-of-text token eos_token_id of each sequence, or at its largest token id with the eos_token_id = 2 of older configs, as in transformers (see CLIPTextModel). The activation hidden_act is "quick_gelu" in OpenAI's checkpoints and "gelu" (exact) in others, such as LAION's.

clip_vision_model(; hidden_size = 768, intermediate_size = 3072, projection_dim = 512,
    num_hidden_layers = 12, num_attention_heads = 12, num_channels = 3,
    image_size = 224, patch_size = 32, hidden_act = "quick_gelu",
    layer_norm_eps = 1e-5, attention_dropout = 0.0, initializer_range = 0.02,
    initializer_factor = 1.0)

Port of transformers' CLIPVisionModel, the vision encoder of CLIP, built from the fields of a CLIPVisionConfig (projection_dim is unused): a ViT over the patches of patch_size × patch_size pixels of square images of image_size pixels, with a class token, a norm before the encoder (pre_layrnorm, as transformers spells it) and one of the class token after it (see CLIPVisionModel).

clip_text_model_with_projection(; vocab_size = 49408, hidden_size = 512,
    intermediate_size = 2048, projection_dim = 512, num_hidden_layers = 12,
    num_attention_heads = 8, max_position_embeddings = 77, hidden_act = "quick_gelu",
    layer_norm_eps = 1e-5, attention_dropout = 0.0, initializer_range = 0.02,
    initializer_factor = 1.0, eos_token_id = 49407)

Port of transformers' CLIPTextModelWithProjection, built from the fields of a CLIPTextConfig: the text encoder clip_text_model, whose pooled features are projected to projection_dim (text_embeds, see CLIPTextModelWithProjection).

clip_vision_model_with_projection(; hidden_size = 768, intermediate_size = 3072,
    projection_dim = 512, num_hidden_layers = 12, num_attention_heads = 12,
    num_channels = 3, image_size = 224, patch_size = 32, hidden_act = "quick_gelu",
    layer_norm_eps = 1e-5, attention_dropout = 0.0, initializer_range = 0.02,
    initializer_factor = 1.0)

Port of transformers' CLIPVisionModelWithProjection, built from the fields of a CLIPVisionConfig: the vision encoder clip_vision_model, whose pooled features are projected to projection_dim (image_embeds, see CLIPVisionModelWithProjection).

CLIPProcessor(image_processor, tokenizer)

transformers' processor of CLIP: a CLIPImageProcessor and the tokenizer of the repository, HuggingFaceTokenizers.jl's Tokenizer, which create_processor loads from a repository such as "openai/clip-vit-base-patch32". The tokenizer needs using HuggingFaceTokenizers before create_processor; without it, tokenizer is nothing, and the processor takes images but no text.

(processor::CLIPProcessor)(; text = nothing, images = nothing)

The inputs of a CLIPModel from text and images (one image or a vector of them, see CLIPImageProcessor), as a NamedTuple with the keys of transformers' CLIPProcessor, in Julia's layout (PyTorch's dimensions reversed), which the model takes as it is (model(inputs)), and so do its encoders alone:

text is a string or a vector of strings, which the tokenizer encodes with its special tokens; one string is a batch of one text. A batch of texts is padded on the right to the longest, with the padding token of the tokenizer, as transformers pads it with padding = true; the other keywords of transformers' tokenizer (truncation, max_length) aren't ported, and texts longer than the max_position_embeddings of the model (77 tokens) are an error of the model.

Zero-shot classification is the softmax of the model's logits_per_image over the texts, the labels in a prompt such as "a photo of a cat":

using Metalhead, HuggingFaceTokenizers
using Flux: softmax
model = create_model("openai/clip-vit-base-patch32")
proc = create_processor("openai/clip-vit-base-patch32")
image = rand(UInt8, 640, 480, 3)                     # width × height × channels
labels = ["cat", "dog", "car"]
inputs = proc(; text = ["a photo of a $l" for l in labels], images = image)
probabilities = softmax(model(inputs).logits_per_image; dims = 1)   # 3 × 1

CLIPImageProcessor(; do_resize = true, size = (; shortest_edge = 224), resample = 3,
                   do_center_crop = true, crop_size = (; height = 224, width = 224),
                   do_rescale = true, rescale_factor = 1 / 255, do_normalize = true,
                   image_mean = (0.48145466, 0.4578275, 0.40821073),
                   image_std = (0.26862954, 0.26130258, 0.27577711),
                   do_convert_rgb = true, default_to_square = false, do_pad = false)

transformers' image processor of CLIP, with the defaults of its Python class; the image processor of a CLIPProcessor, with the settings of a repository such as "openai/clip-vit-base-patch32". Called on one image or a vector of images, it returns the input of the vision encoder of CLIP, which takes it as it is (model(ip(images))), as a NamedTuple:

An image is an array of size (width, height, channels) (torch's (channels, height, width) reversed) of integers from 0 to 255 (an 8-bit image, as UInt8) or of reals from 0 to 1. Each image is

1. converted to RGB with do_convert_rgb: a grey image (a matrix, or one channel, or two with alpha) has its grey repeated, and the alpha of RGBA is dropped, as PIL does;
2. resized with do_resize, with PIL's filter resample, 2 (bilinear) or 3 (bicubic), antialiased as torchvision resizes it: its shorter side to size.shortest_edge and the longer alike, rounded down, or to size.height and size.width;
3. center-cropped to crop_size.height and crop_size.width with do_center_crop, from offsets rounded down, and padded with zeros where it is smaller;
4. multiplied by rescale_factor with do_rescale, and normalized with image_mean and image_std with do_normalize. An image of reals from 0 to 1 is taken as an 8-bit image already rescaled: do_rescale leaves it as it is, and without it the values are multiplied by 255.

An 8-bit image gives the same pixel_values as transformers, and an image of reals the same to Float32 precision. The settings are checked as the processor is made, and also take the forms that transformers takes: size as an integer, the shorter side (or, with default_to_square, the side of a square); crop_size as an integer, the side of a square; resample as the name of PIL's filter (:bicubic); image_mean and image_std as one value for all channels. The padding of transformers' image processors isn't ported: do_pad is false (or nothing, Python's None).