SAM 3

(model::Sam3Model)(pixel_values, input_ids; attention_mask = nothing,
                   input_boxes = nothing, input_boxes_labels = nothing)

Detect and segment the objects of the images pixel_values (width, height, channels, batch) that the text prompts input_ids (seqlen, batch), 1-based token indices padded on the right after the ones of `attentionmask, describe; withinputboxes` (4, numboxes, batch), boxes (cx, cy, w, h) normalized to the image, also those like the boxes labelled 1 in input_boxes_labels (numboxes, batch) and unlike those labelled 0, -10 marking padding (by default, all are positive). The output of [`getvisionfeatures`](@ref Metalhead.Transformers.getvisionfeatures) can replace the images, and that of [`gettextfeatures`](@ref Metalhead.Transformers.gettext_features) the token indices.

Returns a NamedTuple of

• pred_masks (W, H, num_queries, batch): the mask logits of the queries, at 4 times the resolution of the patch grid;
• pred_boxes (4, num_queries, batch): their boxes (x1, y1, x2, y2), normalized;
• pred_logits (num_queries, batch): their classification logits;
• presence_logits (1, batch): the logit that the prompt describes something in the image, whose sigmoid times that of pred_logits gives the scores of the queries;
• semantic_seg (W, H, 1, batch): the logits of a semantic segmentation of the prompt.

(model::Sam3Model)(inputs::NamedTuple)

The model on the output of a Sam3Processor: its pixel_values, input_ids, attention_mask, and input_boxes and input_boxes_labels if it has them; other keys (original_sizes) are ignored.

Sam3VideoModel

transformers' Sam3VideoModel, the model of the checkpoints of SAM 3 (jetjodh/sam3), which detects objects in each frame with its detector_model, a Sam3Model, and tracks them across frames. Only the detector is ported: the model is called and post-processed as its detector, on images.

(model::Sam3VisionModel)(pixel_values)

The vision encoder of SAM 3 on the images pixel_values (width, height, channels, batch): a NamedTuple of the last_hidden_state of the ViT (hidden, W * H, batch), and the fpn_hidden_states and fpn_position_encoding of its neck, vectors of feature maps (W, H, fpnhiddensize, batch) at the scale_factors of the patch grid, and of their sine position encodings (W, H, fpnhiddensize, 1).

Sam3Processor(image_processor, tokenizer; point_pad_value = -10)

transformers' processor of SAM 3: a Sam3ImageProcessor and the tokenizer of the repository, HuggingFaceTokenizers.jl's Tokenizer, which create_processor loads from a repository such as "unsloth/sam3.1" (or "jetjodh/sam3", whose Sam3VideoProcessor it loads as a Sam3Processor). The tokenizer needs using HuggingFaceTokenizers before create_processor; without it, tokenizer is nothing, and the processor takes images but no prompts.

(processor::Sam3Processor)(; images = nothing, text = nothing, input_boxes = nothing,
                           input_boxes_labels = nothing, original_sizes = nothing)

The inputs of a Sam3Model from images (one image or a vector of them, see Sam3ImageProcessor), text and box prompts, as a NamedTuple with the keys of transformers' Sam3Processor, in Julia's layout (PyTorch's dimensions reversed), which the model takes as it is (model(inputs)):

• text is a string or a vector of strings, which the tokenizer encodes with its special tokens. Prompts are padded on the right to 32 tokens, as processing_sam3.py writes, with the padding token of the tokenizer, and longer ones are kept whole. Without text, box prompts have the text "visual", as in transformers, and an entry nothing of a vector is "visual" where its image has boxes.
• input_boxes holds the boxes (x1, y1, x2, y2) of each image in its pixels, a vector of boxes per image (or nothing for none), and input_boxes_labels their labels, 1 (positive, the default) or 0 (negative). Images with fewer boxes are padded with point_pad_value. The boxes need the images, or their original_sizes, (; height, width) or (height, width).

Post-process the outputs of the model with post_process_instance_segmentation, post_process_object_detection or post_process_semantic_segmentation of the processor.

using Metalhead, HuggingFaceTokenizers
model = create_model("unsloth/sam3.1")
proc = create_processor("unsloth/sam3.1")
image = rand(UInt8, 640, 480, 3)                     # width × height × channels
inputs = proc(; images = image, text = "a cat")
objects = post_process_instance_segmentation(proc, model(inputs); threshold = 0.5,
    target_sizes = inputs.original_sizes)
boxes = proc(; images = image, input_boxes = [[(10, 20, 300, 400)]])  # the text "visual"

Sam3ImageProcessor(; do_resize = true, size = (; height = 1008, width = 1008),
                   resample = 2, do_rescale = true, rescale_factor = 1 / 255,
                   do_normalize = true, image_mean = (0.5, 0.5, 0.5),
                   image_std = (0.5, 0.5, 0.5), do_convert_rgb = true,
                   do_center_crop = false, do_pad = false)

transformers' image processor of SAM 3, with the defaults of its Python class; the image processor of a Sam3Processor, with the settings of a repository. Called on one image or a vector of images, it returns a NamedTuple:

An image is an array of size (width, height, channels) (torch's (channels, height, width) reversed) of integers from 0 to 255 (an 8-bit image, as UInt8) or of reals from 0 to 1. Each image is

1. converted to RGB with do_convert_rgb: a grey image (a matrix, or one channel, or two with alpha) has its grey repeated, and the alpha of RGBA is dropped, as PIL does;
2. resized to size with do_resize, with PIL's filter resample, 2 (bilinear) or 3 (bicubic), antialiased as torchvision resizes it;
3. multiplied by rescale_factor with do_rescale, and normalized with image_mean and image_std with do_normalize. An image of reals from 0 to 1 is taken as an 8-bit image already rescaled: do_rescale leaves it as it is, and without it the values are multiplied by 255.

An 8-bit image gives the same pixel_values as transformers, and an image of reals the same to Float32 precision. The settings are checked as the processor is made, and also take the forms that transformers takes: size as an integer, the side of a square; resample as the name of PIL's filter (:bilinear); image_mean and image_std as one value for all channels. The center crop and the padding of transformers' image processors aren't ported: do_center_crop and do_pad are false (or nothing, Python's None).