API reference

create_model(name; pretrained, revision = nothing, subfolder = "", eltype = Float32, kwargs...)

Build a ported model.

name is a Hub repository ("timm/resnet50.a1_in1k", "HuggingFaceTB/SmolLM2-135M"), a bare architecture ("resnet50", "LlamaForCausalLM"; see list_models), or a local directory containing a config.json and model.safetensors.

A repository or a directory names a checkpoint, whose weights are loaded: pretrained defaults to true for them. A bare architecture names the architecture alone, whose weights are random and for which nothing is downloaded: pretrained defaults to false, and pretrained = true loads the weights of its default validated checkpoint. pretrained = false builds the architecture of a repository or directory with random weights, downloading only its config.json. revision selects a branch, tag or commit of the repository; validated checkpoints default to the commit they were validated at. The remaining keyword arguments are the original library's model arguments, e.g. timm's num_classes, or the fields of a transformers model's config.json, e.g. num_hidden_layers. Changing num_classes of a pretrained model re-initialises its classifier.

A diffusers repository holds a pipeline, described by its model_index.json, whose components are in subfolders: create_model builds the whole pipeline, and with subfolder (e.g. "vae" or "transformer") a single component, which "org/name/vae" also names.

A sentence-transformers repository, with a modules.json, holds an embedding model, the modules of a pipeline over a transformers model: create_model builds a SentenceTransformer, whatever the config.json of its transformers model names, and its keyword arguments are the settings of the model, such as truncate_dim.

The model is returned in test mode; call Flux.trainmode! before training.

list_models(; pretrained = false, byfamily = false)

Names of the ported architectures, or with pretrained = true the Hub repositories of the checkpoints they were validated against.

By default, returns a Vector{String}, ordered by architecture name. With byfamily = true, returns a Dict{String, Vector{String}} mapping family identifiers to lists in the same order. For timm, the identifiers are its model module names, such as "resnet", "vision_transformer" and "xception_aligned". With pretrained = true, families without validated checkpoints are omitted.

families = list_models(; byfamily = true)
families["xception_aligned"]
list_models(; byfamily = true, pretrained = true)["resnet"]

forward_features(model, x)

Run model up to its last feature map, before pooling and classification (timm's forward_features).

This and forward_head, forward_intermediates and feature_info are the interface of image backbones and classifiers, and of language models.

forward_head(model, features; pre_logits = false)

Apply the pooling and classifier of model to the output of forward_features. With pre_logits = true, return the pooled features before the classifier.

forward_intermediates(model, x; indices = nothing, norm = false, stop_early = false,
                      output_fmt = "NCHW", intermediates_only = false)

Run model and collect the outputs of its stages or blocks, for dense prediction heads (timm's forward_intermediates). The intermediates are those of feature_info.

• indices: nothing takes all intermediates, an integer k the last k, and a collection the given positions (1-based, unlike timm's).
• norm: apply the final norm of the model to the intermediates, where timm does.
• stop_early: skip the stages after the last selected one.
• output_fmt: "NCHW" returns WHCN feature maps (the Julia layout of PyTorch's NCHW); vision transformers also accept "NLC", for (C, L, N) tokens.

Returns (features, intermediates), where features is the output of forward_features (truncated with stop_early), or only intermediates with intermediates_only = true. Vision transformers take return_prefix_tokens = true, which returns each intermediate as a (spatial, prefix) tuple, where prefix holds the class and register tokens.

feature_info(model) -> Vector{NamedTuple}

The intermediates of forward_intermediates, as timm's model.feature_info: for each, its number of channels num_chs, its reduction (stride) relative to the input, and the PyTorch name module_name of the module that outputs it (timm's module). For a language model, those of its decoder layers: hidden_size channels, a reduction of 1.

create_processor(name; revision = nothing, subfolder = "", kwargs...)

The processor of a checkpoint: it turns raw inputs (images, text, boxes) into the inputs of the model, and the model's outputs back into results (boxes, masks, text), as the checkpoint's configuration and its Python library do. create_model makes the forward pass match PyTorch; the processor makes the whole path from raw input to result match.

name is a Hub repository, a local directory or a bare architecture, as for create_model, and validated repositories default to the commit they were validated at. A branch is read at the commit of the first file found, so that the files come from one commit; offline or without network, from the cache. The settings are read from the repository's files over the defaults of the Python class, strictly: a setting that the port doesn't implement is an error, and an unknown key a warning. The keyword arguments override them by name, e.g. size = (; height = 512), where a NamedTuple merges into a nested setting; each goes to the one component that has a setting of that name, and a name that no component has is an error.

Returns the most complete processor the repository describes, as transformers' AutoProcessor does: the class that its files or its model type name, such as a processor that combines an image processor and a tokenizer (Transformers.Sam3Processor); else its tokenizer, as a TextProcessor; else its image processor, such as the Timm.EvalTransform of every timm checkpoint. A diffusers pipeline and a sentence-transformers model prepare their inputs themselves, and have none. Calling a processor returns what the model takes, a NamedTuple (an array for Timm.EvalTransform), with token indices that start at 1:

using HuggingFaceTokenizers
model = create_model("unsloth/sam3.1")
proc = create_processor("unsloth/sam3.1")
img = rand(UInt8, 640, 480, 3)                       # width × height × channels
inputs = proc(; images = img, text = "a cat")
outputs = model(inputs)
post_process_instance_segmentation(proc, outputs; target_sizes = inputs.original_sizes)

Images are arrays of size (width, height, channels), of integers from 0 to 255 or of reals from 0 to 1, or, with ImageCore loaded, matrices of colorants (see image_array). The processor classes are ported along with their models, and a class that isn't ported is an error. show prints each setting and where it comes from.

Metalhead ports no tokenizer: the tokenizer of a repository is HuggingFaceTokenizers.jl's Tokenizer of its tokenizer.json, which needs using HuggingFaceTokenizers before create_processor. Its token ids start at 0, the token indices of the models and of processors at 1, and decode(proc, indices) decodes token indices:

using HuggingFaceTokenizers
model = create_model("HuggingFaceTB/SmolLM2-135M")
tok = create_processor("HuggingFaceTB/SmolLM2-135M")
logits = model(tok("The capital of France is"))
decode(tok, argmax(logits[:, end]))   # " Paris"

TextProcessor(tokenizer; model_input_names = ["input_ids", "attention_mask"])

The processor of a repository whose processor is its tokenizer, such as that of a language model or a text encoder, where transformers' AutoProcessor returns the tokenizer: HuggingFaceTokenizers.jl's Tokenizer of the repository's tokenizer.json, which create_processor loads after using HuggingFaceTokenizers, with the padding token of its tokenizerconfig.json. Its one setting, `modelinputnames, is that of transformers' tokenizer: the keys that it returns, in order, from"inputids","tokentypeids"and"attentionmask". It is themodelinputnames` of tokenizerconfig.json, else that of the tokenizer_class there (BertTokenizer returns token_type_ids too), else ["input_ids", "attention_mask"].

(processor::TextProcessor)(text; text_pair = nothing, add_special_tokens = true)
(processor::TextProcessor)(; text, text_pair = nothing, add_special_tokens = true)

The inputs of a language model or text encoder from text, a string or a vector of strings, which the tokenizer encodes, with the special tokens that it adds around them unless add_special_tokens = false, as a NamedTuple that the model takes as it is (model(inputs)). text_pair, of the same form and size, encodes each text with a second one, as transformers' tokenizer(text, text_pair) does with the template of the tokenizer ([CLS] a [SEP] b [SEP] for BERT, <s> a </s></s> b </s> for XLM-RoBERTa): the input of a cross-encoder, such as a reranker, for a query and a passage.

One text gives vectors (seq_len,), so that the logits of its next token are model(inputs)[:, end]. A batch of texts is padded on the right to the longest, with the padding token of the tokenizer, as transformers pads it with padding = true; a tokenizer without a padding token pads with the token index 1, which the attention_mask keeps out of the attention, so that the outputs of each text are those it has alone. The texts are not truncated, as transformers' tokenizers by default. token_type_ids are the segment ids of transformers, 0 and 1, not token indices.

Token indices start at 1, the token ids of the tokenizer at 0. decode(processor, indices) (HuggingFaceTokenizers' decode, which takes the keywords of the tokenizer's, such as skip_special_tokens) decodes token indices, and processor.tokenizer is the tokenizer, which takes and returns token ids.

using Metalhead, HuggingFaceTokenizers
model = create_model("HuggingFaceTB/SmolLM2-135M")
tok = create_processor("HuggingFaceTB/SmolLM2-135M")
logits = model(tok("The capital of France is"))
decode(tok, argmax(logits[:, end]))   # " Paris"

reranker = create_model("BAAI/bge-reranker-base")
proc = create_processor("BAAI/bge-reranker-base")
reranker(proc(["what is a panda?", "what is a panda?"];
    text_pair = ["The giant panda is a bear.", "Paris is in France."]))   # 1 × 2 scores

Timm.EvalTransform(; input_size = (3, 224, 224), interpolation = "bicubic",
                   mean = (0.485, 0.456, 0.406), std = (0.229, 0.224, 0.225),
                   crop_pct = 0.875, crop_mode = "center")

timm's evaluation transform, create_transform(**resolve_data_config(...)): the image processor of every timm checkpoint, which create_processor builds from the pretrained_cfg of its config.json, resolved as timm's resolve_data_config resolves it: a setting that the configuration doesn't have, or that is false in Python ("", [], 0), takes the default above.

Called on one image or a vector of images, it returns the input of the model, not a NamedTuple: a Float32 array of size (W, H, 3, batch), where (3, H, W) is input_size, of the images resized, cropped and normalized, bit for bit as timm transforms them as PIL images. One image gives a batch of one.

model = create_model("timm/convnext_tiny.fb_in1k")
transform = create_processor("timm/convnext_tiny.fb_in1k")
logits = model(transform(image))    # transform(image): 224 × 224 × 3 × 1

An image is an array of size (width, height, channels) (torch's (channels, height, width) reversed) of integers from 0 to 255 (an 8-bit image, as UInt8) or of reals from 0 to 1, which are converted to 8 bits as torchvision's ToPILImage converts them, or, with ImageCore loaded, a matrix of colorants (see image_array). Grey images, and the alpha of RGBA, are converted to RGB as Pillow's convert("RGB") does. Each image is

1. resized with Pillow's filter interpolation ("nearest", "bilinear", "bicubic", "box", "hamming" or "lanczos"): its short side to floor(input_size / crop_pct) with crop_mode = "center", both sides with crop_mode = "squash". (A non-square input_size with "center" resizes with "bilinear", as in timm.) timm's third crop mode, "border", is not supported;
2. center-cropped to the height and width of input_size (channels, height, width), padded with zeros where it is smaller;
3. scaled to [0, 1] and normalized with mean and std.

The settings also take the forms that timm's create_transform takes: input_size as (height, width) or an integer, the side of a square; mean and std as one value for all channels; interpolation and crop_mode as symbols; and crop_pct and crop_mode as nothing for their defaults.

create_processor(repo; use_test_size = true) resolves test_input_size and test_crop_pct in place of input_size and crop_pct, where the configuration has them, as resolve_data_config(use_test_size = True) does: many checkpoints are evaluated at a larger size than the one they were trained at. The other keywords of create_processor override the settings, e.g. crop_pct = 1.0. Timm.pretrained_cfg returns the pretrained_cfg as it is.

Qwen3VLProcessor(image_processor, tokenizer)

transformers' processor of Qwen3-VL and Qwen3.5: a Qwen2VLImageProcessor and the tokenizer of the repository, HuggingFaceTokenizers.jl's Tokenizer, which create_processor loads from a repository such as "Qwen/Qwen3.5-0.8B". The tokenizer needs using HuggingFaceTokenizers before create_processor; without it, tokenizer is nothing, and the processor takes images but no text.

(processor::Qwen3VLProcessor)(; images = nothing, text = nothing)

The inputs of a Qwen3.5 vision-language model (qwen3_5_for_conditional_generation) from images (one image or a vector of them, see Qwen2VLImageProcessor) and text, as a NamedTuple with the keys of transformers' Qwen3VLProcessor, in Julia's layout (PyTorch's dimensions reversed), which the model takes as it is (model(inputs)): the outputs of the tokenizer and of the image processor, merged.

where D = 3 * temporal_patch_size * patch_size^2 (1536 for Qwen3.5).

text is a string or a vector of strings, which the tokenizer encodes with its special tokens. One text gives vectors (seq_len,), as the model takes them, so that the logits of its next token are model(inputs)[:, end]. A batch of texts is padded on the right to the longest, with the padding token of the tokenizer, as transformers pads it with padding = true; the other keywords of transformers' tokenizer (truncation, max_length) aren't ported.

Each image placeholder of the texts, <|image_pad|>, stands for the next image, in order through the texts, and becomes as many image tokens as the image has merged patches, prod(image_grid_thw[:, i]) ÷ merge_size^2, as in transformers. The numbers of placeholders and of images must be equal (transformers fails on more placeholders than images, and its model on fewer). The chat template isn't applied: write the placeholders where it would, <|vision_start|><|image_pad|><|vision_end|>.

using Metalhead, HuggingFaceTokenizers
model = create_model("Qwen/Qwen3.5-0.8B")
proc = create_processor("Qwen/Qwen3.5-0.8B")
image = rand(UInt8, 640, 480, 3)                       # width × height × channels
prompt = "<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>" *
         "Describe this image.<|im_end|>\n<|im_start|>assistant\n" *
         "<think>\n\n</think>\n\n"
inputs = proc(; images = image, text = prompt)        # 300 image tokens for 30 × 40 patches
logits = model(inputs)                                 # vocab_size × seq_len
HuggingFaceTokenizers.decode(proc.tokenizer, [argmax(logits[:, end]) - 1])

Qwen2VLImageProcessor(; do_resize = true, resample = 3,
                      size = (; shortest_edge = 3136, longest_edge = 1003520),
                      do_rescale = true, rescale_factor = 1 / 255, do_normalize = true,
                      image_mean = (0.48145466, 0.4578275, 0.40821073),
                      image_std = (0.26862954, 0.26130258, 0.27577711),
                      do_convert_rgb = true, patch_size = 14, temporal_patch_size = 2,
                      merge_size = 2)

transformers' image processor of Qwen2-VL, with the defaults of its Python class; the image processor of a Qwen3VLProcessor, with the settings of a repository such as "Qwen/Qwen3.5-0.8B" (patches of 16 pixels, mean and standard deviation 0.5). The keys min_pixels and max_pixels of Qwen2-VL's configurations, or of create_processor, set size.shortest_edge and size.longest_edge, as in transformers. Called on one image or a vector of images, it returns the inputs of the vision encoder of Qwen3.5 (qwen3_5_vision_model), which takes them as they are (model(ip(images))), as a NamedTuple in Julia's layout (PyTorch's dimensions reversed):

where D = 3 * temporal_patch_size * patch_size^2 (1536 for Qwen3.5). The columns of pixel_values (the rows of transformers') are the patches of the images one after the other, each image ordered by windows of merge_size × merge_size patches, row-major over the windows and within each window. A patch is flattened as an array of size (patch_size, patch_size, temporal_patch_size, 3) (width first), its pixels repeated temporal_patch_size times: an image is a video of one frame.

An image is an array of size (width, height, channels) (torch's (channels, height, width) reversed) of integers from 0 to 255 (an 8-bit image, as UInt8) or of reals from 0 to 1. Each image is

1. converted to RGB with do_convert_rgb: a grey image (a matrix, or one channel, or two with alpha) has its grey repeated, and the alpha of RGBA is dropped, as PIL does;
2. resized with do_resize, with PIL's filter resample, 2 (bilinear) or 3 (bicubic), antialiased as torchvision resizes it, to the size whose sides are multiples of patch_size * merge_size, with an area from size.shortest_edge to size.longest_edge pixels, that keeps the aspect ratio of the image best (transformers' smart_resize). Without do_resize, the sides of an image must be multiples of patch_size * merge_size;
3. multiplied by rescale_factor with do_rescale, and normalized with image_mean and image_std with do_normalize. An image of reals from 0 to 1 is taken as an 8-bit image already rescaled: do_rescale leaves it as it is, and without it the values are multiplied by 255;
4. cut into patches of patch_size × patch_size pixels.

An 8-bit image gives the same pixel_values as transformers, and an image of reals the same to Float32 precision. The settings are checked as the processor is made, and also take the forms that transformers takes: resample as the name of PIL's filter (:bicubic); image_mean and image_std as one value for all channels.

post_process_object_detection(processor, outputs; kwargs...)

The objects that a model detects in each image, from its outputs and its processor (transformers' post_process_object_detection): for each image, a NamedTuple of their scores and boxes. The processor of each model has a method; that of SAM 3 is Sam3Processor.

post_process_instance_segmentation(processor, outputs; kwargs...)

The objects that a model segments in each image, from its outputs and its processor (transformers' post_process_instance_segmentation): for each image, a NamedTuple of their scores, boxes and masks. The processor of each model has a method; that of SAM 3 is Sam3Processor.

post_process_semantic_segmentation(processor, outputs; kwargs...)

The semantic segmentation of each image by a model, from its outputs and its processor (transformers' post_process_semantic_segmentation). The processor of each model has a method; that of SAM 3 is Sam3Processor.

image_array(image) -> AbstractArray{<:Real, 3}

image as processors take it: an array of size (width, height, channels) (torch's (channels, height, width) reversed) of integers from 0 to 255 (an 8-bit image, such as UInt8) or of reals from 0 to 1. A matrix is a grey image of one channel.

With ImageCore loaded (as by Images.jl), it also takes a matrix of colorants of size (height, width), such as the Matrix{RGB{N0f8}} that FileIO.load returns, as permutedims(channelview(img), (3, 2, 1)) but with the channels of any colorant in the order R, G, B (then alpha), or gray (then alpha). N0f8 values, those of 8-bit images, become the UInt8 values they store, and others reals from 0 to 1.

image_matrix(x) -> AbstractMatrix{<:Colorant}

The inverse of image_array: the array x of size (width, height, channels), such as an image that a pipeline generates or a mask of SAM 3, as a matrix of colorants of size (height, width), which FileIO saves and image viewers show. It needs ImageCore (or Images.jl) loaded.

The channels make RGB (3), RGBA (4), gray (1) or gray and alpha (2) colorants; a matrix (width, height), such as a mask, a gray image. 8-bit values (integers from 0 to 255) make N0f8 colorants, Bools gray Bools, and other reals, from 0 to 1, Float32 colorants. An array of size (width, height, channels, batch) gives a vector of images.

using ImageCore, FileIO
image = pipe("A photo of a red fox sitting in fresh snow"; num_inference_steps = 4)
FileIO.save("fox.png", Metalhead.image_matrix(cpu(image))[1])

Metalhead.Timm.pretrained_cfg(name; revision = nothing) -> NamedTuple

Preprocessing settings of a timm checkpoint (mean, std, input_size, crop_pct, interpolation, ...), as stored in its config.json. name is a Hub repository, a bare architecture (for its default checkpoint) or a local directory, as for Metalhead.create_model. input_size is in PyTorch order (C, H, W). Metalhead.create_processor returns the transform they describe, a Timm.EvalTransform.

(model::Qwen3_5ForConditionalGeneration)(inputs::NamedTuple)
(model::Qwen3_5Model)(inputs::NamedTuple)

The model on the output of a Qwen3VLProcessor: its input_ids and attention_mask, and pixel_values and image_grid_thw if it has images; other keys (mm_token_type_ids) are ignored. The input_ids of one text, a vector, give the logits (or hidden states) of size (vocab_size, seq_len), so that logits[:, end] are those of the next token. A batch is padded on the right, as the processor pads it, and each text gets the outputs it gets alone, at the positions of its tokens.

(model::Qwen3_5VisionModel)(pixel_values, grid_thw) -> (; last_hidden_state, pooler_output)

Encode images and videos, given as patches: pixel_values of size (in_channels * temporal_patch_size * patch_size^2, num_patches), and grid_thw, an integer matrix of size (3, num_images) whose column (t, h, w) gives the numbers of patches of an image along time, height and width (see qwen3_5_vision_model). Returns the hidden states of the patches, of size (hidden_size, num_patches), and the merged tokens of the pooler_output, of size (out_hidden_size, num_patches ÷ spatial_merge_size^2), image after image.

(model::Qwen3_5VisionModel)(inputs::NamedTuple)

The model on the output of a Qwen2VLImageProcessor (or of a Qwen3VLProcessor with images): its pixel_values and image_grid_thw; other keys are ignored.

Flux2KleinPipeline

diffusers' pipeline of FLUX.2 [klein], which generates images from text, or edits images (reference images, image) as the text says, built by create_model from a repository such as "black-forest-labs/FLUX.2-klein-4B". Its components are text_encoder, a Qwen3 language model whose hidden states condition the image; transformer, a Flux2Transformer2DModel that predicts the flow of the latents; vae, the AutoencoderKLFlux2 that decodes them (and encodes the reference images); scheduler, the FlowMatchEulerDiscreteScheduler; and tokenizer, the tokenizer of the text encoder: HuggingFaceTokenizers.jl's Tokenizer of the tokenizer/ subfolder. Without using HuggingFaceTokenizers before create_model, it is a reference to the tokenizer.json, which the pipeline loads once HuggingFaceTokenizers is loaded, without loading the pipeline again; it is nothing for a pipeline saved without a tokenizer, which takes prompt embeddings only. is_distilled checkpoints (FLUX.2 [klein] 4B, 9B) sample in a few steps without classifier-free guidance; the base ones (FLUX.2 [klein] base 4B, 9B) take about 50 steps and a guidance_scale of 4.

gpu(pipeline) and bf16(pipeline) move and convert every component but the tokenizer.

using Metalhead, Flux, HuggingFaceTokenizers
using CUDA, cuDNN

pipe = create_model("black-forest-labs/FLUX.2-klein-4B"; eltype = BFloat16)
pipe = gpu(pipe)

image = pipe("A cat holding a sign that says hello world"; num_inference_steps = 4)
edited = pipe("Make the sign say goodbye"; image = image[:, :, :, 1], num_inference_steps = 4)

encode_prompt(pipeline, prompt; max_sequence_length = 512,
              text_encoder_out_layers = (9, 18, 27))
encode_prompt(pipeline, input_ids; attention_mask = nothing,
              text_encoder_out_layers = (9, 18, 27))

The embeddings of a prompt (a string) or of a vector of prompts, as diffusers' encode_prompt makes them, of size (length(text_encoder_out_layers) * hidden_size, max_sequence_length, batch). Each prompt is formatted with the chat template of the Qwen3 text encoder (a user message, with thinking disabled and a generation prompt), tokenized with pipeline.tokenizer (which needs using HuggingFaceTokenizers), truncated to max_sequence_length tokens and padded on the right to them. The embeddings are the hidden states of the layers text_encoder_out_layers of the text encoder (transformers' output_hidden_states, of which that of the last layer is normed), concatenated along the features.

The second method takes the prompts tokenized: input_ids, 1-based token indices of size (seq_len, batch), padded on the right as attention_mask says, and moved to the device of the text encoder.

encode_prompt(pipeline::ZImagePipeline, prompt; max_sequence_length = 512)
encode_prompt(pipeline::ZImagePipeline, input_ids; attention_mask = nothing)

The embeddings of a prompt (a string) or of a vector of prompts, as diffusers' ZImagePipeline.encode_prompt makes them: a vector with a matrix of size (hidden_size, length) for each prompt, the hidden states of its tokens in the second-to-last layer of the text encoder (transformers' output_hidden_states[-2]), without the padding. Each prompt is formatted with the chat template of the Qwen3 text encoder (a user message and a generation prompt, with thinking enabled), tokenized with pipeline.tokenizer (which needs using HuggingFaceTokenizers) and truncated to max_sequence_length tokens.

The second method takes the prompts tokenized: input_ids, 1-based token indices of size (seq_len, batch), padded on the right as attention_mask says.

(transformer::Flux2Transformer2DModel)(hidden_states, encoder_hidden_states, timestep,
                                       img_ids, txt_ids, guidance = nothing)

The velocity predicted by the FLUX.2 transformer for the image tokens hidden_states of size (in_channels, image_seq_len, batch), conditioned on the text tokens encoder_hidden_states of size (joint_attention_dim, text_seq_len, batch), at the timesteps (a vector of batch values in [0, 1], the noise levels σ). img_ids and txt_ids are the integer coordinates of the tokens, of size (4, seq_len) (or (4, seq_len, batch), of which the first is used), from which the rotary embeddings are computed. guidance is the guidance scale of guidance-distilled checkpoints (guidance_embeds = true). Returns an array of size (patch_size^2 * out_channels, image_seq_len, batch).

ZImagePipeline

diffusers' text-to-image pipeline of Z-Image, built by create_model from a repository such as "Tongyi-MAI/Z-Image-Turbo". Its components are text_encoder, a Qwen3 decoder (a LlamaModel, transformers' Qwen3Model) whose hidden states condition the image; transformer, a ZImageTransformer2DModel that predicts the flow of the latents; vae, the AutoencoderKL that decodes them; scheduler, the FlowMatchEulerDiscreteScheduler; and tokenizer, the tokenizer of the text encoder, as for Flux2KleinPipeline: HuggingFaceTokenizers.jl's Tokenizer of the tokenizer/ subfolder, a reference to it that the pipeline loads once HuggingFaceTokenizers is loaded, or nothing without one.

The defaults of the pipeline are those of diffusers, for Z-Image: 50 steps with a guidance_scale of 5. Z-Image-Turbo is distilled to sample in 9 steps without guidance (guidance_scale = 0).

gpu(pipeline) and bf16(pipeline) move and convert every component but the tokenizer.

using Metalhead, Flux, HuggingFaceTokenizers
pipe = create_model("Tongyi-MAI/Z-Image-Turbo"; eltype = BFloat16)
image = pipe("A cat holding a sign that says hello world"; num_inference_steps = 9,
             guidance_scale = 0)

(transformer::ZImageTransformer2DModel)(x, t, cap_feats; patch_size = 2,
                                        f_patch_size = 1)

The output of the Z-Image transformer for the latent images x, of size (width, height, in_channels, batch) or (width, height, frames, in_channels, batch), at the timesteps t (a vector of batch values in [0, 1]: 1 - σ for the noise level σ), conditioned on the captions cap_feats: a vector of batch matrices of size (cap_feat_dim, length), the prompt embeddings of each image without their padding, or an array of size (cap_feat_dim, length, batch) when they have the same length. Returns an array of the size of x, the opposite of the velocity of the flow: Z-Image pipelines step along its negation.

As in diffusers, the patches of each image and the tokens of its caption are padded to a multiple of 32 with the learned x_pad_token and cap_pad_token, and the captions of a batch are then padded to the same length, which the attention masks. patch_size and f_patch_size select one of the pairs of all_patch_size and all_f_patch_size.

encode(vae, x) -> DiagonalGaussianDistribution

The posterior of the latents of the images x, of size (width, height, 3, batch) with values in [-1, 1]: mode gives the latents, of size (width ÷ f, height ÷ f, latent_channels, batch) for the downsampling factor f.

decode(vae, z)

The images, with values about [-1, 1], of the latents z.

DiagonalGaussianDistribution(parameters)

diffusers' posterior of an autoencoder: a Gaussian with the mean and the log-variance (clamped to [-30, 20]) that are the two halves of the channels of the encoder's output parameters. mode returns its mean, and sample draws from it.

mode(posterior::DiagonalGaussianDistribution)

The mean of the posterior of an autoencoder.

sample([rng], posterior::DiagonalGaussianDistribution)

A draw from the posterior of an autoencoder, with noise from rng (generated on the CPU).

FlowMatchEulerDiscreteScheduler

diffusers' Euler sampler of flow-matching models, as configured by the scheduler_config.json of a pipeline. Unlike diffusers' scheduler it keeps no state: set_timesteps returns the schedule, and step takes the noise levels of a step.

set_timesteps(scheduler, num_inference_steps; sigmas = nothing, mu = nothing) -> (timesteps, sigmas)

The Float32 timesteps of the denoising steps (sigmas .* num_train_timesteps) and their noise levels, with a final 0. The sigmas are shifted by mu with use_dynamic_shifting, which then requires it, or else by the scheduler's shift. By default they are evenly spaced from 1 to the smallest noise level of training.

step(scheduler, model_output, sigma, sigma_next, sample)

The sample of an Euler step of the flow model_output from the noise level sigma to sigma_next, computed in at least Float32 and returned in the element type of model_output.

compute_empirical_mu(image_seq_len, num_steps)

The shift of the noise levels of FLUX.2 for image_seq_len image tokens and num_steps steps, fitted by Black Forest Labs.

convmixer(; dim, depth, kernel_size = 9, patch_size = 7, num_classes = 1000, kwargs...)

Port of timm's ConvMixer. Keyword arguments are those of the timm class, with act_layer given by name ("gelu" or "relu"); kernel_size and patch_size are integers or (height, width) tuples.

convnext(; depths, dims, num_classes = 1000, in_chans = 3, kwargs...)

Port of timm's ConvNeXt with the patch stem and the default head (ConvNeXt, ConvNeXt-Atto to Nano). Keyword arguments are those of the timm class.

create_conv2d(in_chs, out_chs, kernel_size; stride, dilation, padding = "", groups, depthwise, bias = false, num_experts = 0)

timm.layers.createconv2d. Sizes are Ints or torch-order (h, w) pairs; a vector of kernel sizes gives a MixedConv2d and `numexperts > 0a [CondConv2d](@ref).paddingis""(PyTorch-style symmetric),"same"(TF-style, aConv2dSameunless it is static),"valid"`, or an explicit torch padding.

decode_arch_def(arch_def, depth_multiplier = 1.0; depth_trunc = "ceil",
                experts_multiplier = 1, fix_first_last = false, group_size = nothing)

Decode the block definition strings of the stages of arch_def (a vector of vectors of strings) into a vector of vectors of block arguments. depth_multiplier is a number or one per stage; experts_multiplier scales the number of CondConv experts.

densenet(; growth_rate = 32, block_config = (6, 12, 24, 16), kwargs...)

Port of timm's DenseNet (DenseNet-BC, with the optional deep stem and blur pooling). Keyword arguments are those of the timm class, except that aa_layer is "blurpool2d" or nothing. memory_efficient is accepted and ignored: it only changes how gradients are computed.

efficientnet(; block_args, num_classes = 1000, num_features = 1280, kwargs...)

Port of timm's EfficientNet class (EfficientNet, EfficientNet-V2, MobileNet-V1/V2, MNASNet and the others built by timm.models.efficientnet). block_args comes from decode_arch_def; norm_layer is nothing or the arguments of BatchNorm2d, and se_layer(channels; act_layer, rd_ratio) creates the squeeze-and-excitation modules. Other keyword arguments are those of the timm class.

inception_resnet_v2(; num_classes = 1000, in_chans = 3, drop_rate = 0.0)

Port of timm's InceptionResnetV2. Keyword arguments are those of the timm class.

inception_v3(; num_classes = 1000, in_chans = 3, drop_rate = 0.0)

Port of timm's InceptionV3 without the auxiliary classifier (aux_logits = false). Keyword arguments are those of the timm class. The input must be at least 75x75.

inception_v4(; num_classes = 1000, in_chans = 3, drop_rate = 0.0)

Port of timm's InceptionV4. Keyword arguments are those of the timm class.

mlp_mixer(; img_size = 224, patch_size = 16, num_blocks = 8, embed_dim = 512, kwargs...)

Port of timm's MlpMixer: MLP-Mixer, ResMLP, gMLP and the GLU mixer. Keyword arguments are those of the timm class, with the classes it takes given by name: block_layer is "MixerBlock", "ResBlock" or "SpatialGatingBlock", mlp_layer is "Mlp", "GluMlp" or "GatedMlp" (only the pairs of timm's entry points), norm_layer is "layernorm" (eps = 1e-6) or "affine" and act_layer is "gelu" or "silu". init_values is the layer scale of ResBlock (functools.partial(ResBlock, init_values = ...) in timm). global_pool is "avg" or "", which applies the head to every token.

mobilenetv3(; block_args, num_classes = 1000, num_features = 1280, kwargs...)

Port of timm's MobileNetV3 class (MobileNet-V3, LCNet, MobileNet-V4): the EfficientNet stem and blocks with the "efficient head", where global pooling comes before the head convolution and there is no norm layer before the classifier (but one after the head convolution with head_norm). num_features is the width of the head convolution. block_args comes from decode_arch_def, norm_layer is nothing or the arguments of BatchNorm2d, and se_layer(channels; act_layer, rd_ratio) creates the squeeze-and-excitation modules. Other keyword arguments are those of the timm class.

resnet(; block, layers, num_classes = 1000, in_chans = 3, kwargs...)

Port of timm's ResNet (ResNet, ResNeXt, Wide ResNet, SE-ResNet, ECA-ResNet, Res2Net and the c/d/s/t stem variants). Keyword arguments are those of the timm class. block_args is a NamedTuple of keywords for the blocks: attn_layer = "se" or "eca" for BasicBlock and Bottleneck, scale for Bottle2neck.

vgg(; cfg, num_classes = 1000, in_chans = 3, norm_layer = nothing, kwargs...)

Port of timm's VGG. cfg lists the output channels of the 3x3 convolutions, with "M" for a max pool, and norm_layer = "batchnorm2d" adds batch normalisation (the _bn variants). Other keyword arguments are those of the timm class.

vit(; img_size = 224, patch_size = 16, embed_dim = 768, depth = 12, num_heads = 12, kwargs...)

Port of timm's VisionTransformer, including the options of DINOv2 (init_values, reg_tokens, no_embed_class), CLIP (pre_norm, act_layer = "quick_gelu") and SigLIP (class_token = false, global_pool = "map"), and dynamic_img_size. Keyword arguments are those of the timm class, except that the epsilon of norm_layer is given as norm_eps (timm's partial(LayerNorm, eps = ...)). act_layer is "gelu" (the default), "gelu_tanh" or "quick_gelu". img_size and patch_size are integers or (height, width) tuples, as in timm. Other options (qk_norm, custom layers, ...) error.

xception_aligned(; block_cfg, num_classes = 1000, in_chans = 3, kwargs...)

Port of timm's XceptionAligned (Aligned Xception, with the pre-activation variant). Keyword arguments are those of the timm class, except that norm_eps stands for the eps of a norm_layer = partial(nn.BatchNorm2d, eps = ...). block_cfg is a sequence of named tuples (in_chs, out_chs, stride) with the optional no_skip and start_with_relu.

chunk_gated_delta_rule(q, k, v, g, beta; chunk_size = 64)

transformers' torch_chunk_gated_delta_rule with use_qk_l2norm_in_kernel = true, for the queries and keys q and k (headkdim, heads, seqlen, batch), the values v (headvdim, heads, seqlen, batch), the log decays g and the rates beta (heads, seqlen, batch). For each head, with q and k normalised and q scaled by `headkdim^-0.5, the stateS` (headvdim, headk_dim) starts at zero and at each token becomes S = exp(g) * S then S += beta * (v - S * k) * kᵀ, and the output is S * q.

As in transformers, the sequence is padded to chunks of chunk_size tokens, computed in parallel but for the state carried from one chunk to the next, in at least Float32. Each chunk solves a unit triangular system, here by inverting its matrix by blocks, which runs on a GPU and is differentiable. The output has the element type of v.

gemma3_for_causal_lm(; vocab_size = 262208, hidden_size = 2304, num_hidden_layers = 26, kwargs...)

Port of transformers' Gemma3ForCausalLM, the Gemma 3 text model, built from the fields of its Gemma3TextConfig with the same names and defaults, and rope_theta, rope_local_base_freq, rope_scaling and sliding_window_pattern, which older config.jsons have in place of rope_parameters and layer_types. It takes and returns the arrays of llama_for_causal_lm.

Gemma 3 differs from Llama in its decoder layers: the token embeddings are multiplied by √hidden_size, the RMSNorms scale by 1 + weight, the outputs of the attention and of the MLP are normalised too, the queries and keys are normalised per head and scaled by query_pre_attn_scalar^-0.5, the MLP gate is gelu_pytorch_tanh, and the logits may be softcapped (final_logit_softcapping). The layers attend either to all the previous tokens ("full_attention") or to the last sliding_window of them ("sliding_attention"), as layer_types says, with rotary embeddings of their own: rope_parameters has one for each layer type. Bidirectional attention (use_bidirectional_attention) errors. attn_logit_softcapping is ignored, as transformers ignores it.

get_image_features(model::CLIPModel, pixel_values; interpolate_pos_encoding = false)
get_image_features(model::CLIPModel, inputs::NamedTuple;
                   interpolate_pos_encoding = false)

The output of the vision encoder of CLIP on the images pixel_values (width, height, channels, numimages), or on the `pixelvaluesof the output of its processor: itslasthiddenstateand, aspooleroutput` (projectiondim, numimages), its pooled features projected by `visualprojection, not normalised (transformers'getimagefeatures). See [CLIPVisionModel](@ref Metalhead.Transformers.CLIPVisionModel) forinterpolateposencoding`.

get_image_features(model::Siglip2Model, pixel_values; spatial_shapes,
                   pixel_attention_mask = nothing)
get_image_features(model::Siglip2Model, inputs::NamedTuple)

The output of the vision model of SigLIP 2 on images given as patches (see Siglip2VisionModel), or on the output of its image processor: (; last_hidden_state, pooler_output), whose pooler_output (hidden, numimages) are the embeddings of the images before their normalization, as transformers' `getimage_features` returns them.

get_image_features(model::SiglipModel, pixel_values; interpolate_pos_encoding = false)
get_image_features(model::SiglipModel, inputs::NamedTuple)

The output of the vision model of SigLIP on the images pixel_values (width, height, channels, numimages), or on the `pixelvaluesof the output of its processor:(; lasthiddenstate, pooleroutput)(see [SiglipVisionModel](@ref Metalhead.Transformers.SiglipVisionModel)), whosepooleroutput(hidden, num_images) are the embeddings of the images before their normalization, as transformers'getimagefeatures` returns them.

get_text_features(model::CLIPModel, input_ids; attention_mask = nothing)
get_text_features(model::CLIPModel, inputs::NamedTuple)

The output of the text encoder of CLIP on the token indices input_ids (seqlen, numtexts), 1-based and padded on the right after the ones of attention_mask, or on the input_ids (and attention_mask) of the output of its processor: its last_hidden_state and, as pooler_output (projectiondim, numtexts), its pooled features projected by text_projection, not normalised (transformers' get_text_features).

get_text_features(model::Sam3Model, input_ids; attention_mask = nothing)

The features of the text prompts of SAM 3: the last_hidden_state of its text encoder on the token indices input_ids (seqlen, batchsize), 1-based and padded on the right after the ones of attention_mask, and its projection to the hidden size of the detector (pooler_output, as transformers names it). Pass them to the model in place of the token indices to reuse them for several images.

get_text_features(model::Union{SiglipModel, Siglip2Model}, input_ids;
                  attention_mask = nothing)
get_text_features(model::Union{SiglipModel, Siglip2Model}, inputs::NamedTuple)

The output of the text model of SigLIP or SigLIP 2 on the token indices input_ids (seqlen, numtexts), 1-based, or on the input_ids (and attention_mask) of the output of its processor: (; last_hidden_state, pooler_output) (see SiglipTextModel), whose pooler_output (projectionsize, numtexts) are the embeddings of the texts before their normalization, as transformers' get_text_features returns them.

get_vision_features(model::Sam3Model, pixel_values)

The output of the vision encoder of SAM 3 on the images pixel_values (see Sam3VisionModel). Pass it to the model in place of the images to reuse it for several prompts.

granite_for_causal_lm(; vocab_size = 32000, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' GraniteForCausalLM (IBM Granite 3 and 4.1), built from the fields of its GraniteConfig with the same names and defaults, rope_theta and rope_scaling, which older config.jsons have in place of rope_parameters, and head_dim, which the attention reads where a config.json has it. It takes and returns the arrays of llama_for_causal_lm.

Granite is Llama with four multipliers: the token embeddings are multiplied by embedding_multiplier, the attention scores by attention_multiplier (in place of 1/√head_dim), the outputs of the attention and of the MLP by residual_multiplier, and the logits are divided by logits_scaling.

granitemoe_for_causal_lm(; vocab_size = 32000, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' GraniteMoeForCausalLM (IBM Granite 3 MoE), built from the fields of its GraniteMoeConfig with the same names and defaults, and the keys of granite_for_causal_lm that it adds (rope_theta, rope_scaling, head_dim). It takes and returns the arrays of llama_for_causal_lm.

Granite MoE is Granite whose MLP is a sparse mixture of num_local_experts gated MLPs of intermediate_size features: each token goes through the num_experts_per_tok experts of largest router logit, whose outputs are weighted by the softmax of these logits. The training options (output_router_logits, router_aux_loss_coef) are not supported.

granitemoehybrid_for_causal_lm(; vocab_size = 32000, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' GraniteMoeHybridForCausalLM (IBM Granite 4.0) without its Mamba layers: the models whose layer_types are all "attention" (or "full_attention"), such as granite-4.0-350m and granite-4.0-1b. A layer is that of granitemoeshared_for_causal_lm, whose shared expert is always present, with no mixture of experts if num_local_experts = 0. The attention has a rotary embedding only with position_embedding_type = "rope". Mamba layers ("mamba" or "linear_attention", the default) error, and the mamba_* keys of config.json are ignored.

granitemoeshared_for_causal_lm(; vocab_size = 32000, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' GraniteMoeSharedForCausalLM: Granite MoE (see granitemoe_for_causal_lm) with a shared expert, a gated MLP of shared_intermediate_size features that every token goes through, unless shared_intermediate_size = 0.

llama_for_causal_lm(; vocab_size = 32000, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' LlamaForCausalLM, built from the fields of its LlamaConfig (the config.json of a checkpoint), with the same names and defaults. create_model ignores the other keys of config.json, such as the token ids and generation settings, as transformers does. The rotary embedding is the default one or, with rope_scaling or rope_parameters, one with rope_type = "linear" or "llama3" (Llama 3.1 and later); other types error.

The model maps token indices of size (seq_len, batch_size), or a vector for a single sequence, to logits of size (vocab_size, seq_len, batch_size), with causal attention. Token indices start at 1: add 1 to the ids of a Hugging Face tokenizer. A batch of sequences of different lengths, padded on the right, takes the tokenizer's attention_mask of size (seq_len, batch_size) as a keyword argument, which keeps the padding out of the keys. The output of its processor, a TextProcessor, can replace both: model(tok(texts)). forward_features returns the hidden states after the final norm, and forward_intermediates those of each layer.

moe_experts(gate_up, down, act, x, top_k_index, top_k_weights)

The experts of a sparse mixture of experts, as transformers' MixtralExperts: each token, a column of x (hidden, n), goes through its experts top_k_index (k, n), whose outputs down * (act(gate) .* up) are summed with the weights top_k_weights (k, n). The stacked weights of the experts are torch's gate_up_proj (numexperts, 2 * intermediate, hidden) and `downproj(num_experts, hidden, intermediate), sogateup` has size (hidden, 2 * intermediate, numexperts) and down (intermediate, hidden, num_experts).

The tokens are grouped by expert and padded to the largest group, so that each projection is one batched_mul, and each token gathers back the outputs of its experts. Unlike a scatter, this needs no atomic additions, which are nondeterministic, and slow in BFloat16 on GPUs before sm_90.

qwen2_for_causal_lm(; vocab_size = 151936, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' Qwen2ForCausalLM, built from the fields of its Qwen2Config with the same names and defaults, and head_dim, which Qwen2Config lacks but transformers reads. transformers derives Qwen2 from Llama's classes, adding a bias to the query, key and value projections, so the model is a LlamaForCausalLM (see llama_for_causal_lm for its inputs and outputs). Sliding-window attention (use_sliding_window, layer_types) and rotary embeddings other than the default, "linear" and "llama3" ones error.

qwen3_5_decode(model::Qwen3_5TextModel, inputs_embeds, position_ids; attention_mask = nothing)

The hidden states (hiddensize, seqlen, batchsize) after the final norm of the Qwen3.5 text model for the input embeddings `inputsembeds(hidden_size, seq_len, batch_size), transformers'Qwen35TextModel.forward(inputsembeds=...).positionids` are the temporal, height and width positions of the tokens for the rotary embedding, an integer array (3, seqlen, batchsize) or (3, seqlen, 1) of 0-based positions (torch's position_ids (3, batchsize, seqlen) permuted), or nothing for 0:seq_len-1 on each row, those of text. attention_mask (seqlen, batchsize) marks the tokens of a batch padded on the right, as for the token indices.

qwen3_5_for_causal_lm(; vocab_size = 248320, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' Qwen3_5ForCausalLM, the text model of Qwen3.5: the decoder of qwen3_5_text_model, with its keywords, and the output projection lm_head, tied to the token embeddings with tie_word_embeddings. It takes and returns the arrays of llama_for_causal_lm, and the keyword position_ids of qwen3_5_decode. The multi-token prediction weights (mtp.*) of the checkpoints are not loaded, as in transformers.

qwen3_5_for_conditional_generation(; text_config, vision_config, image_token_id = 248056,
                                   tie_word_embeddings = false)

Port of transformers' Qwen3_5ForConditionalGeneration, the vision-language model of the Qwen3.5 checkpoints, built from its Qwen3_5Config: the vision encoder qwen3_5_vision_model of vision_config, the decoder qwen3_5_text_model of text_config (both NamedTuples of their config's fields; other keys are ignored) and the output projection lm_head, tied to the token embeddings with tie_word_embeddings. Videos and the multi-token prediction weights (mtp.*) of the checkpoints are not supported.

The model maps token indices of size (seq_len, batch_size), or a vector, to logits of size (vocab_size, seq_len, batch_size), as llama_for_causal_lm does, and takes the keyword arguments attention_mask and, for images, pixel_values and image_grid_thw, as made by the image processor of a checkpoint (Qwen2VLImageProcessor). The token indices then hold, for each image in order, a run of prod(image_grid_thw[:, i]) ÷ 4 image tokens (image_token_id + 1, <|image_pad|>), between the tokens of <|vision_start|> and <|vision_end|>, as the chat template of the checkpoints writes them. The merged tokens of the image replace the image tokens, and the multimodal rotary embedding gives them the positions of their grid. The processor of a checkpoint, a Qwen3VLProcessor, makes all of these from images and a prompt with a placeholder for each image, and the model takes its output as it is: model(proc(; images, text)).

qwen3_5_moe_for_causal_lm(; vocab_size = 248320, hidden_size = 2048, num_hidden_layers = 40, kwargs...)

Port of transformers' Qwen3_5MoeForCausalLM, the text model of Qwen3.5 MoE (Qwen3.8-2.4T-A95B): the decoder of qwen3_5_moe_text_model, with its keywords, and the output projection lm_head, tied to the token embeddings with tie_word_embeddings. It takes and returns the arrays of qwen3_5_for_causal_lm. The multi-token prediction weights (mtp.*) of the checkpoints are not loaded, as in transformers.

qwen3_5_moe_for_conditional_generation(; text_config, vision_config, image_token_id = 248056,
                                       tie_word_embeddings = false)

Port of transformers' Qwen3_5MoeForConditionalGeneration, the vision-language model of the Qwen3.5 MoE and Qwen3.6 MoE checkpoints (Qwen3.6-35B-A3B), built from its Qwen3_5MoeConfig: the model of qwen3_5_for_conditional_generation, with its inputs, outputs and processor, whose decoder is qwen3_5_moe_text_model of text_config. Videos and the multi-token prediction weights (mtp.*) of the checkpoints are not supported.

qwen3_5_moe_text_model(; vocab_size = 248320, hidden_size = 2048, num_hidden_layers = 40, kwargs...)

Port of transformers' Qwen3_5MoeTextModel, the decoder of Qwen3.5 MoE, built from the fields of its Qwen3_5MoeTextConfig with the same names and defaults, and full_attention_interval, rope_theta, rope_scaling and partial_rotary_factor, which the config reads from config.json. It is the decoder of qwen3_5_text_model, with its inputs and outputs, whose MLPs are sparse mixtures of num_experts gated MLPs of moe_intermediate_size features with a shared expert: each token goes through the num_experts_per_tok experts of largest router probability, weighted by these probabilities renormalised to sum to 1, and through the shared expert, a gated MLP of shared_expert_intermediate_size features weighted by a sigmoid gate. The training options (output_router_logits, router_aux_loss_coef) are ignored.

qwen3_5_text_model(; vocab_size = 248320, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' Qwen3_5TextModel, the decoder of Qwen3.5, built from the fields of its Qwen3_5TextConfig with the same names and defaults, and full_attention_interval, rope_theta, rope_scaling and partial_rotary_factor, which the config reads from config.json. It maps token indices of size (seq_len, batch_size), or a vector, to the hidden states after the final norm, and takes the keywords attention_mask and position_ids (see qwen3_5_decode, which takes input embeddings instead).

Its layers have either a gated delta rule ("linear_attention", Qwen3_5GatedDeltaNet) or causal attention ("full_attention"), as layer_types says, by default every full_attention_interval-th layer. The attention has a sigmoid gate on its output, norms its queries and keys per head and rotates the first head_dim * partial_rotary_factor features of each with the multimodal rotary embedding of rope_parameters["mrope_section"] (only the default rope_type). The RMSNorms scale by 1 + weight. The output of the gated delta rule is gated with SiLU, as in transformers, which ignores output_gate_type: it may be nothing, "silu" or "swish" (Qwen3.8's), and other values are an error.

qwen3_5_vision_model(; depth = 27, hidden_size = 1152, num_heads = 16, patch_size = 16,
                     spatial_merge_size = 2, temporal_patch_size = 2, kwargs...)

Port of transformers' Qwen3_5VisionModel, the vision encoder of Qwen3.5, built from the fields of its Qwen3_5VisionConfig (the vision_config of a Qwen3.5 checkpoint) with the same names and defaults. deepstack_visual_indexes, which config.json may carry, must be empty, as transformers' Qwen3.5 has no deepstack mergers; the rotary embedding is the axial one (rope_type = "axial", or "default").

The model takes (pixel_values, grid_thw), as the image processor of a checkpoint (Qwen2VLImageProcessor) makes them, whose output it also takes as it is (model(ip(images))), and returns the NamedTuple (; last_hidden_state, pooler_output):

• pixel_values, of size (in_channels * temporal_patch_size * patch_size^2, num_patches), is the transpose of transformers' pixel_values: each column is a patch, flattened as torch's (in_channels, temporal_patch_size, patch_size, patch_size) (that is, as a Julia array of size (patch_size, patch_size, temporal_patch_size, in_channels), width first).
• grid_thw, an integer matrix of size (3, num_images) (transformers' grid_thw transposed), has a column (t, h, w) for each image or video: its number of patches along time (frames of temporal_patch_size), height and width. The patches of the images follow one another; within an image, frame after frame, each frame is ordered by windows of spatial_merge_size × spatial_merge_size patches, row-major over the windows and row-major within each window.
• last_hidden_state has size (hidden_size, num_patches). pooler_output, the merged tokens that the language model takes, image after image, has size (out_hidden_size, num_patches ÷ spatial_merge_size^2).

Each frame of each image attends to its own patches only.

qwen3_for_causal_lm(; vocab_size = 151936, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' Qwen3ForCausalLM, built from the fields of its Qwen3Config with the same names and defaults. transformers derives Qwen3 from Llama's classes, with a Qwen3Attention that normalises each query and key head, so the model is a LlamaForCausalLM (see llama_for_causal_lm for its inputs and outputs). Sliding-window attention (use_sliding_window, layer_types) and rotary embeddings other than the default, "linear" and "llama3" ones error.

qwen3_moe_for_causal_lm(; vocab_size = 151936, hidden_size = 2048, num_hidden_layers = 24, kwargs...)

Port of transformers' Qwen3MoeForCausalLM (Qwen3-30B-A3B, Qwen3-235B-A22B), built from the fields of its Qwen3MoeConfig with the same names and defaults, and head_dim, rope_theta and rope_scaling, which transformers reads from config.json. It takes and returns the arrays of llama_for_causal_lm.

Qwen3 MoE is Qwen3 whose MLP is a sparse mixture of num_experts gated MLPs of moe_intermediate_size features: each token goes through the num_experts_per_tok experts of largest router probability, weighted by these probabilities, renormalised to sum to 1 if norm_topk_prob. The layers in mlp_only_layers (from 0), and all if num_experts = 0, have a dense MLP of intermediate_size features instead, and so do those whose number (from 1) isn't a multiple of decoder_sparse_step. config.json files written by transformers 5 name num_experts num_local_experts. Sliding-window attention (use_sliding_window) and rotary embeddings other than the default, "linear" and "llama3" ones error. The training options (output_router_logits, router_aux_loss_coef) are ignored.

sam3_model(; vision_config = nothing, text_config = nothing,
           geometry_encoder_config = nothing, detr_encoder_config = nothing,
           detr_decoder_config = nothing, mask_decoder_config = nothing)

Port of transformers' Sam3Model, the detector of SAM 3 (Carion et al., 2025), built from its Sam3Config: NamedTuples of the fields of the configs of its parts (other keys are ignored), each nothing for the defaults of transformers. The vision encoder sam3_vision_model of vision_config and the CLIP text encoder clip_text_model_with_projection of text_config feed a DETR encoder and decoder, whose queries the mask decoder turns into masks; box prompts go through a geometry encoder.

The model maps images and text prompts, and optionally box prompts, to the masks, boxes and scores of the objects they describe: see Sam3Model for its inputs and outputs, and Sam3Processor, which create_processor loads, for its inputs and to read its outputs.

sam3_video_model(; detector_config = nothing, tracker_config = nothing)

Port of the detector of transformers' Sam3VideoModel (see Sam3VideoModel), built from its Sam3VideoConfig: the sam3_model of detector_config, a NamedTuple of the fields of its Sam3Config. The tracker of tracker_config is not ported, and the weights of tracker_model and tracker_neck are not loaded.

sam3_vision_model(; backbone_config = nothing, fpn_hidden_size = 256,
                  scale_factors = (4.0, 2.0, 1.0, 0.5))

Port of transformers' Sam3VisionModel, the vision encoder of SAM 3, from the fields of its Sam3VisionConfig: the ViT sam3_vit_model of backbone_config (a NamedTuple of its config's fields; other keys are ignored), and a neck that resamples its patch grid by each of scale_factors into feature maps of fpn_hidden_size channels.

sam3_vit_model(; hidden_size = 1024, intermediate_size = 4736, num_hidden_layers = 32,
    num_attention_heads = 16, num_channels = 3, image_size = 1008, patch_size = 14,
    hidden_act = "gelu", layer_norm_eps = 1e-6, attention_dropout = 0.0,
    window_size = 24, global_attn_indexes = (7, 15, 23, 31),
    layer_scale_init_value = nothing, pretrain_image_size = 336, hidden_dropout = 0.0,
    initializer_range = 0.02, rope_parameters = nothing, rope_theta = nothing)

Port of transformers' Sam3ViTModel, the ViT of SAM 3, from the fields of its Sam3ViTConfig: its layers attend within windows of window_size × window_size patches, but for those of global_attn_indexes (0-based), which attend over the whole grid of an image of image_size, the size the model takes. A 2D axial rotary embedding (rope_theta, from rope_parameters too) gives the positions, scaled to those of a window in the global layers; the learned position embeddings of the pretrain_image_size grid are tiled over larger grids. As in transformers, layer_scale_init_value is not used.

siglip2_model(; text_config = nothing, vision_config = nothing,
              initializer_factor = 1.0)

Port of transformers' Siglip2Model, the image-text dual encoder of the NaFlex checkpoints of SigLIP 2 (see Siglip2Model), built from its Siglip2Config: the siglip_text_model of text_config and the siglip2_vision_model of vision_config, NamedTuples of the fields of their configs (other keys are ignored), each nothing for the defaults of transformers. The projection_size of the text model must be the hidden_size of the vision model.

siglip2_vision_model(; hidden_size = 768, intermediate_size = 3072,
    num_hidden_layers = 12, num_attention_heads = 12, num_channels = 3,
    patch_size = 16, num_patches = 256, hidden_act = "gelu_pytorch_tanh",
    layer_norm_eps = 1e-6, attention_dropout = 0.0, vision_use_head = true)

Port of transformers' Siglip2VisionModel, the vision encoder of the NaFlex checkpoints of SigLIP 2 (see Siglip2VisionModel), built from the fields of its Siglip2VisionConfig: SigLIP's vision transformer, whose patches, flattened, are embedded by a Dense layer, with num_patches learned position embeddings of a square grid, resized to the grid of each image. vision_use_head = false is not supported.

siglip_model(; text_config = nothing, vision_config = nothing, initializer_factor = 1.0)

Port of transformers' SiglipModel, the image-text dual encoder of SigLIP and of the SigLIP 2 checkpoints of fixed resolution (see SiglipModel), built from its SiglipConfig: the siglip_text_model of text_config and the siglip_vision_model of vision_config, NamedTuples of the fields of their configs (other keys are ignored), each nothing for the defaults of transformers. The projection_size of the text model must be the hidden_size of the vision model.

siglip_text_model(; vocab_size = 32000, hidden_size = 768, intermediate_size = 3072,
    num_hidden_layers = 12, num_attention_heads = 12, max_position_embeddings = 64,
    hidden_act = "gelu_pytorch_tanh", layer_norm_eps = 1e-6, attention_dropout = 0.0,
    projection_size = nothing)

Port of transformers' SiglipTextModel, the text encoder of SigLIP and SigLIP 2 (see SiglipTextModel), built from the fields of its SiglipTextConfig (or Siglip2TextConfig, the same): a transformer whose tokens attend to each other in both directions, and whose last token is projected by head to projection_size features (hidden_size by default).

siglip_vision_model(; hidden_size = 768, intermediate_size = 3072,
    num_hidden_layers = 12, num_attention_heads = 12, num_channels = 3,
    image_size = 224, patch_size = 16, hidden_act = "gelu_pytorch_tanh",
    layer_norm_eps = 1e-6, attention_dropout = 0.0, vision_use_head = true)

Port of transformers' SiglipVisionModel, the vision encoder of SigLIP (see SiglipVisionModel), built from the fields of its SiglipVisionConfig: a vision transformer of the patches of patch_size × patch_size pixels of images of image_size × image_size, with learned position embeddings and no class token, whose tokens an attention-pooling head pools. vision_use_head = false, which the configurations of some vision-language models carry, is not supported.

forward_intermediates(model::Union{LlamaForCausalLM, Gemma3ForCausalLM, GraniteForCausalLM, Qwen3_5ForCausalLM},
                      input_ids; indices = nothing, norm = false, stop_early = false,
                      intermediates_only = false, attention_mask = nothing)

The outputs of the decoder layers of a language model (Llama, Qwen2, Qwen3, Qwen3.5, Gemma 3, Granite) for the token indices input_ids of size (seq_len, batch_size), each of size (hidden_size, seq_len, batch_size), or of a vector for a single sequence, each of size (hidden_size, seq_len): the i-th intermediate is transformers' output_hidden_states[i], the output of the i-th layer, except that the last one is not normed unless norm = true. The features are the hidden states after the final norm, as forward_features returns them. indices, norm, stop_early and intermediates_only are those of the vision models, and attention_mask that of the model, as is the position_ids keyword of Qwen3.5. The output of a processor can replace input_ids and attention_mask, as for the model. feature_info gives the hidden_size and the module of each layer.

autoencoder_kl_flux2(; latent_channels = 32, block_out_channels = (128, 256, 512, 512), kwargs...)

Port of diffusers' AutoencoderKLFlux2, the autoencoder of FLUX.2, built from the arguments of its constructor (the config.json of the vae of a pipeline), with the same names and defaults. Only DownEncoderBlock2D and UpDecoderBlock2D blocks are supported, which all checkpoints use.

The model maps images of size (width, height, in_channels, batch) to images of the same size, through the mean of the posterior: see encode and decode. Its bn holds the mean and the variance of the latents in patches of patch_size, with which FLUX.2 pipelines normalise them; it takes no part in the model's forward pass.

flux2_transformer_2d_model(; num_layers = 8, num_single_layers = 48, num_attention_heads = 48,
                           attention_head_dim = 128, kwargs...)

Port of diffusers' Flux2Transformer2DModel, the transformer of FLUX.2, built from the arguments of its constructor (the config.json of the transformer of a pipeline), with the same names and defaults. The defaults are those of FLUX.2 [dev]; FLUX.2 [klein] 4B has 5 double- and 20 single-stream blocks of 24 heads.

See Flux2Transformer2DModel for its inputs and output.

autoencoder_kl(; latent_channels = 4, block_out_channels = (64,),
               scaling_factor = 0.18215, shift_factor = nothing, kwargs...)

Port of diffusers' AutoencoderKL, the variational autoencoder of Stable Diffusion, FLUX.1 and Z-Image, built from the arguments of its constructor (the config.json of the vae of a pipeline), with the same names and defaults. Only DownEncoderBlock2D and UpDecoderBlock2D blocks are supported, which all checkpoints use.

The model maps images of size (width, height, in_channels, batch) to images of the same size, through the mean of the posterior: see encode and decode. Pipelines decode z / scaling_factor .+ shift_factor from the latents z they denoise, which the model holds as its fields scaling_factor and shift_factor (nothing for none). sample_size, latents_mean, latents_std and force_upcast are settings of tiling and of other pipelines, which the port ignores.

z_image_transformer_2d_model(; dim = 3840, n_layers = 30, n_heads = 30,
                             cap_feat_dim = 2560, kwargs...)

Port of diffusers' ZImageTransformer2DModel, the transformer of Z-Image, built from the arguments of its constructor (the config.json of the transformer of a pipeline), with the same names and defaults, which are those of Z-Image (Turbo has axes_lens = (1536, 512, 512)). As in diffusers, n_kv_heads is unused. The Omni variant (siglip_feat_dim) is not supported.

See ZImageTransformer2DModel for its inputs and output.

Metalhead.Timm

Flux ports of timm models. Each ported model mirrors the PyTorch module tree, so that pretrained state_dicts load by name; build them with Metalhead.create_model.

EfficientNetBuilder(; output_stride = 32, pad_type = "", round_chs_fn = round_channels,
                    se_from_exp = false, act_layer = nothing, norm_layer = BATCHNORM2D,
                    aa_layer = nothing, se_layer = nothing, drop_path_rate = 0.0,
                    layer_scale_init_value = nothing)

Builds the stages of blocks from decoded block arguments. Calling it as builder(in_chs, block_args) returns a vector of Chains, and leaves the channel count of the last block in builder.in_chs. se_layer(channels; act_layer, rd_ratio) creates the squeeze-and-excitation module of a block. aa_layer (anti-aliasing, see create_aa) is supported by the cn, er, uir and mqa blocks, and layer_scale_init_value applies to the uir and mqa blocks of MobileNetV4.

MixedConv2d(in_chs, out_chs, kernel_sizes; stride, dilation, padding, depthwise, bias)

timm.layers.MixedConv2d (MixConv): the channels are split into one group per kernel size, each convolved with its own kernel size, and concatenated. A ModuleDict in timm, so the convs' keys are 0.weight, 1.weight, ... with no segment for the container.

CondConv2d(in_chs, out_chs, kernel_size; num_experts, stride, dilation, padding, groups, bias)

timm.layers.CondConv2d: a conv whose kernel (and bias) is a per-sample mixture of num_experts kernels, called as m(x, routing_weights) with routing weights of size (num_experts, batch). weight is torch's (num_experts, params) reversed: column e is the flattened kernel of expert e. The samples are convolved at once, as one grouped conv.

Metalhead.Transformers

Flux ports of Hugging Face Transformers models. Each ported model mirrors the PyTorch module tree, so that pretrained state_dicts load by name; build them with Metalhead.create_model from a Hub repository.

Language models take token indices starting at 1, as Flux's Embedding does: add 1 to the ids of a Hugging Face tokenizer, and subtract 1 from the index of a logit to get its token id. With using HuggingFaceTokenizers, Metalhead.create_processor loads the tokenizer of a repository.

MultiheadAttention

PyTorch's nn.MultiheadAttention with batch_first = True, keys and values of the size of the queries and no extra biases, as the attention-pooling heads of SigLIP use it: the queries, keys and values are projected by the blocks of in_proj_weight, of size (embed_dim, 3 * embed_dim) (torch's (3 * embed_dim, embed_dim)), and in_proj_bias. Called on (query, key, value) of size (embed_dim, len, batch), with the keys of each sequence limited to its first kv_lengths, it returns (attn_output, nothing), as torch returns the output and the attention weights, which aren't computed.

Metalhead.Diffusers

Flux ports of Hugging Face Diffusers models and pipelines. Each ported model mirrors the PyTorch module tree, so that pretrained state_dicts load by name. Metalhead.create_model builds a whole pipeline from a repository, such as Flux2KleinPipeline from "black-forest-labs/FLUX.2-klein-4B" or ZImagePipeline from "Tongyi-MAI/Z-Image-Turbo", or one of its components with subfolder.

Images are WHCN arrays with values in [-1, 1] for the autoencoders, and [0, 1] for the output of pipelines. The reference images of a pipeline are images as processors take them (see Metalhead.image_array).

timestep_embedding(timesteps, embedding_dim; flip_sin_to_cos = false,
                   downscale_freq_shift = 1, scale = 1, max_period = 10_000)

diffusers' get_timestep_embedding: the sinusoidal embeddings of a vector of (possibly fractional) timesteps, of size (embedding_dim, length(timesteps)), on the device of timesteps. As in diffusers, they are computed in at least Float32: in BFloat16, the angles of timesteps near 1000 would be off by several radians.

AdaLayerNormContinuous(embedding_dim, conditioning_embedding_dim; elementwise_affine = true,
                       eps = 1e-5, bias = true)

diffusers' adaptive LayerNorm of tokens (embedding_dim, seq_len, batch), scaled and shifted by a projection of the conditioning embedding (conditioning_embedding_dim, batch).

Metalhead.SentenceTransformers

Flux ports of sentence-transformers models, the text embeddings of the Hub: the modules of a repository's modules.json (Transformer, Pooling, Dense, Normalize) over the ported transformers models, with the settings of its configuration files, read strictly. Metalhead.create_model builds a SentenceTransformer from a repository with a modules.json, such as "Qwen/Qwen3-Embedding-0.6B".

ArchSpec(format, name, build; family = "other", defaults = (;),
         references = Pair{String,String}[], ignore = Regex[])

A ported architecture, as config.json files in format name it. build(; kwargs...) constructs the model from the keyword arguments of the original library (e.g. timm's), merged over defaults. references lists repo => sha checkpoints the port was validated against; the first one is the default pretrained checkpoint. ignore lists extra checkpoint keys the model doesn't use. family is the original library's family identifier, used to group architectures in list_models and the documentation.

AbstractConfigFormat

How the config.json of a model repository names an architecture and its arguments: timm's (architecture, model_args, pretrained_cfg), a future library's architectures, ... A ported library defines a subtype and registers its architectures with it. Only architecture_of is required; the other methods have defaults.

architecture_of(format, config) -> Union{Nothing, String}

The name of the architecture described by the parsed config.json, or nothing if the configuration isn't in this format.

model_args(format, spec, config, kwargs::NamedTuple) -> NamedTuple

The keyword arguments for spec.build: the defaults of the architecture, overridden by the configuration and then by the user's kwargs.

skip_prefix(format, config, args) -> Union{Nothing, String}

The prefix of the checkpoint keys that don't load into a model built with args, which keeps its initial values (e.g. the classifier after changing num_classes). Throws if the checkpoint can't be loaded into such a model at all.

checkpoint_prefix(model, ckpt::Checkpoint) -> String

The prefix of the keys under which the checkpoint ckpt holds the weights of model: "" (the default), or the name of a module that some checkpoints of model nest its weights in, and that the original library removes from their keys as it loads them (transformers' PrefixChange: "text_model" for the CLIPTextModel checkpoints saved before transformers 5).

weights_name(format) -> String

The name of the safetensors weights of a model in format: "model" for model.safetensors or its shards (the default).

instantiate(format, spec, source, config; pretrained, eltype, kwargs...)

Build the model spec of the parsed config, from model_args, in the element type eltype, and with pretrained = true load its weights from source. A format whose configuration describes several models (a diffusers pipeline) builds them here.

AbstractProcessor

A processor, as create_processor returns it: an image processor, or a processor that combines processors and a tokenizer. Each is a plain immutable struct whose fields are the attributes of the Python class (its settings, and the processors it combines), with a keyword constructor whose defaults are those of the Python class, and a provenance field that records where each setting comes from, for show.

AbstractImageProcessor <: AbstractProcessor

A processor of images, called as ip(images) on one image or a vector of them (see image_array).

@processor struct T <: AbstractProcessor ... end
@processor (components...,) struct T <: AbstractProcessor ... end

Base.@kwdef for a processor class, whose constructors make the processor from standardize_settings of its settings, so that a processor made by hand fails as early as a loaded one. components names the fields that hold the processors it combines, each of type Union{Nothing, C} with the default nothing, which create_processor loads as a processor of class C from the same repository; a component that isn't a C is an error. The component tokenizer, of type Any, is the tokenizer of the repository: HuggingFaceTokenizers.jl's Tokenizer of its tokenizer.json, or nothing without HuggingFaceTokenizers or without the file. With components, T(components...; kwargs...) also makes it from them, e.g. Sam3Processor(image_processor, tokenizer; point_pad_value = -10).

register_processor!(name, T; aliases = (), model_types = ())

Register the processor class T as the port of transformers' (or another library's) class name, which configurations name in processor_class or image_processor_type, and of the classes aliases. The suffixes Fast and Pil of class names, those of transformers' former image processors and of its PIL backend, are dropped before lookup. model_types are the model_types of config.json whose processor is T (AutoProcessor's PROCESSOR_MAPPING), for repositories whose processor files don't name a class.

ignored_keys(T) -> NamedTuple

The keys of a configuration that the processor class T knows but doesn't read, as key = "reason", e.g. (; mask_size = "only resizes the segmentation maps of training"). Other keys that aren't settings of T warn once. An override of an ignored key is an error that gives the reason.

legacy_keys(T) -> NamedTuple

Former names of settings of the processor class T, as key = path, where path is the name of a setting or the path to an entry of a nested one, e.g. (; min_pixels = (:size, :shortest_edge)). The value of a legacy key replaces the one at its path, in files and overrides, and drops the entry of the legacy key from the nested setting, as transformers does.

standardize_settings(T, settings::NamedTuple) -> NamedTuple

The settings of a processor of class T as its constructor stores them, from settings, the values it is given for the fields that are neither components nor provenance: the forms of a setting that the Python library also takes, converted (an integer size to (; height, width)), and an ArgumentError for any other value, or for one that the port doesn't implement (do_center_crop = true), naming the supported values. An error whose message names a setting in backticks, as Metalhead's errors do, also names the file that set it when the processor is loaded. The default returns settings as they are, which the constructor converts to the types of the fields.

instantiate_processor(format::AbstractConfigFormat, files, config; kwargs...)
    -> Union{Nothing, AbstractProcessor}

The processor of a repository whose config.json, parsed as config, is in format and carries the preprocessing of its checkpoint (timm's pretrained_cfg), with the overrides kwargs; or nothing (the default) for create_processor to find the processor files of the repository as transformers' AutoProcessor does. files reads the other files of the repository.

check_inputs(model, inputs::NamedTuple, required; hint = "")

Throw if the NamedTuple inputs of model(inputs) lacks keys of required, naming them and the keys it has, so that the inputs of another model fail at the call rather than deep inside. Every model(inputs::NamedTuple) method calls it on the keys it can't do without.

pil_resize(img, (width, height), filter = :bicubic; torchvision = false)
    -> Array{UInt8, 3}

img, a (width, height, channels) array of UInt8, resized as Pillow's Image.resize((width, height), filter) resizes an 8-bit image, bit for bit. filter is one of :nearest, :box, :bilinear, :hamming, :bicubic and :lanczos.

With torchvision = true, for :bilinear and :bicubic, it is resized bit for bit as torchvision resizes a uint8 tensor on an x86 CPU, with antialias = True. torch's variant of the algorithm, from Pillow-SIMD, always resizes the width first, and rounds the weights of each pass to as many bits as int16 holds. (For :bilinear without AVX2, and on a GPU, torchvision resizes in floating point instead.)

Checkpoint(paths)

Lazy view over one or more safetensors files (e.g. the shards of a sharded checkpoint). Indexing with a PyTorch state_dict key returns the tensor in Julia layout, i.e. with the PyTorch dimensions reversed (NCHW → WHCN).

load_state!(model, ckpt::Checkpoint; prefix = "", ignore = DEFAULT_IGNORE, skip = nothing, strict = true)

Copy the tensors of ckpt into model, matching PyTorch state_dict keys derived from the model structure (see torch_params). Keys matching a regex in ignore may be absent from the model; parameters whose key starts with skip keep their current values. With strict = true, missing keys, unused checkpoint keys and shape mismatches throw.

torch_params(model; prefix = "") -> (params::Vector{TorchParam}, aliases::Vector{Pair{String,String}})

Enumerate the parameters and buffers of model with their PyTorch state_dict keys. An array reached twice (tied weights) is listed once; later keys are returned as aliases.

TorchParam(key, array, kind)

A parameter or buffer of a Flux model, with its PyTorch state_dict key. kind selects the transform from the checkpoint's Julia-layout tensor: :plain (none), :dense (transpose) or :conv (flip the spatial dimensions).