API reference

Metalhead — Module
Metalhead

Flux ports of the models of the Hugging Face Hub: the vision models of timm, the language, text, vision-language and segmentation models of transformers, the text-to-image pipelines of diffusers and the text embeddings of sentence-transformers. create_model builds a model, with its weights, and create_processor its pre- and post-processing.

source

Everything on the pages of this reference is public, exported or declared with public on Julia 1.11 and later, except the Internals and Porting pages and the sections called Internals. The fields of model structs mirror PyTorch's and are not public, but for the documented generation_config of the language models.

Models

create_model and list_models apply to every ported model. The other functions are the interface of image backbones and classifiers, and of language models, which generate text.

Metalhead.create_model — Function
create_model(name; pretrained, revision = nothing, subfolder = "", eltype = Float32, kwargs...)

Build a ported model.

name is a Hub repository ("timm/resnet50.a1_in1k", "HuggingFaceTB/SmolLM2-135M"), a bare architecture ("resnet50", "LlamaForCausalLM"; see list_models), or a local directory containing a config.json and model.safetensors.

A repository or a directory names a checkpoint, whose weights are loaded: pretrained defaults to true for them. A bare architecture names the architecture alone, whose weights are random and for which nothing is downloaded: pretrained defaults to false, and pretrained = true loads the weights of its default validated checkpoint, which for a timm architecture is the one timm.create_model(name, pretrained=True) loads. pretrained = false builds the architecture of a repository or directory with random weights, downloading only its config.json. revision selects a branch, tag or commit of the repository; validated checkpoints default to the commit they were validated at. The remaining keyword arguments are the original library's model arguments, e.g. timm's num_classes, or the fields of a transformers model's config.json, e.g. num_hidden_layers; for a model with sub-configs (text_config, vision_config, ...) they are NamedTuples that are merged into those of the config.json. Changing num_classes of a pretrained model re-initialises its classifier.

A diffusers repository holds a pipeline, described by its model_index.json, whose components are in subfolders: create_model builds the whole pipeline, and with subfolder (e.g. "vae" or "transformer") a single component, which "org/name/vae" also names.

A sentence-transformers repository, with a modules.json, holds an embedding model, the modules of a pipeline over a transformers model: create_model builds a SentenceTransformer, whatever the config.json of its transformers model names, and its keyword arguments are the settings of the model, such as truncate_dim.

The model is returned in test mode; call Flux.trainmode! before training.

source
Metalhead.list_models — Function
list_models(; pretrained = false, byfamily = false)

Names of the ported architectures, or with pretrained = true the Hub repositories of the checkpoints they were validated against.

By default, returns a Vector{String}, ordered by architecture name. With byfamily = true, returns a Dict{String, Vector{String}} mapping family identifiers to lists in the same order. For timm, the identifiers are its model module names, such as "resnet", "vision_transformer" and "xception_aligned". With pretrained = true, families without validated checkpoints are omitted.

families = list_models(; byfamily = true)
families["xception_aligned"]
list_models(; byfamily = true, pretrained = true)["resnet"]
source
Metalhead.forward_head — Function
forward_head(model, features; pre_logits = false)

Apply the pooling and classifier of model to the output of forward_features. With pre_logits = true, return the pooled features before the classifier.

source
Metalhead.forward_intermediates — Function
forward_intermediates(model, x; indices = nothing, norm = false, stop_early = false,
                      output_fmt = "NCHW", intermediates_only = false)

Run model and collect the outputs of its stages or blocks, for dense prediction heads (timm's forward_intermediates). The intermediates are those of feature_info.

  • indices: nothing takes all intermediates, an integer k the last k, and a collection the given positions (1-based, unlike timm's).
  • norm: apply the final norm of the model to the intermediates, where timm does.
  • stop_early: skip the stages after the last selected one.
  • output_fmt: "NCHW" returns WHCN feature maps (the Julia layout of PyTorch's NCHW); vision transformers also accept "NLC", for (C, L, N) tokens.

Returns (features, intermediates), where features is the output of forward_features (truncated with stop_early), or only intermediates with intermediates_only = true. Vision transformers take return_prefix_tokens = true, which returns each intermediate as a (spatial, prefix) tuple, where prefix holds the class and register tokens.

source
Metalhead.feature_info — Function
feature_info(model) -> Vector{NamedTuple}

The intermediates of forward_intermediates, as timm's model.feature_info: for each, its number of channels num_chs, its reduction (stride) relative to the input, and the PyTorch name module_name of the module that outputs it (timm's module). For a language model, those of its decoder layers: hidden_size channels, a reduction of 1.

source
Metalhead.Transformers.generate — Function
generate(model, input_ids; attention_mask = nothing, generation_config = model.generation_config,
         rng = Random.default_rng(), kwargs...)
generate(model, inputs::NamedTuple; kwargs...)

The tokens that the language model generates after each sequence of the token indices input_ids, of size (seq_len, batch_size) (padded on the right, with its attention_mask), or of a vector for a single sequence, as transformers' model.generate: a matrix of size (n, batch_size), or a vector, of their token indices; the output of a TextProcessor can replace input_ids and attention_mask. These are the new tokens only, those of transformers' generate(...)[:, seq_len:]: [input_ids; generate(model, input_ids)] is the whole sequence. A sequence that ends with an end-of-text token is followed by padding tokens, until the others end too, or n reaches max_new_tokens.

It decodes greedily or samples with the settings of generation_config, that of the checkpoint (see GenerationConfig), and the settings kwargs change them: max_new_tokens, do_sample, temperature, top_k, top_p, min_p, repetition_penalty, min_new_tokens, eos_token_id, ... Sampling draws from rng. The other keyword arguments are the inputs of the model, such as the pixel_values of Qwen3.5.

The models with a KVCache run on the prompts, then on each new token alone; the others (Gemma 3 and Qwen3.5) on the whole sequences at each step. The logits processors and the sampler run on the device of the model; on a GPU, each step copies only the new tokens to the host.

Examples

using Metalhead, HuggingFaceTokenizers

model = create_model("HuggingFaceTB/SmolLM2-135M")
tok = create_processor("HuggingFaceTB/SmolLM2-135M")
new = generate(model, tok("The capital of France is"); max_new_tokens = 16)
decode(tok, new)
generate(model, tok(["Hello,", "The weather"]); do_sample = true, temperature = 0.7)
source
Metalhead.Transformers.GenerationConfig — Type
GenerationConfig(; kwargs...)
GenerationConfig(config::GenerationConfig; kwargs...)

The settings of generate, with the names and defaults of transformers' GenerationConfig, for greedy decoding and multinomial sampling; the second form changes the settings kwargs of config.

  • max_new_tokens: the number of tokens to generate at most; or max_length, the length of the prompt (padded) and the new tokens; neither: 20 new tokens.
  • min_new_tokens, or min_length: until then, the end-of-text tokens are excluded.
  • do_sample = false: the most likely token; true samples the next token from the logits divided by temperature = 1.0, among the top_k = 50 most likely tokens (0: all of them), then the most likely of cumulative probability top_p = 1.0, then those of probability at least min_p times that of the most likely (nothing: all).
  • repetition_penalty = 1.0: divides the positive logits of the tokens that a sequence has, and multiplies the negative ones.
  • eos_token_id: the end-of-text tokens, which end a sequence (an integer, or several); pad_token_id, the token of a sequence after its end (the first end-of-text token by default); bos_token_id. These are ids of a Hugging Face tokenizer, from 0, as in the files: token indices are one more.

The generation_config of a language model that create_model loads is that of its checkpoint, as transformers' model.generation_config: its generation_config.json, or the token ids of its config.json (of its text_config too) without one. A generation_config.json that asks for anything else, such as beam search (num_beams > 1), no_repeat_ngram_size or suppress_tokens, is an error.

source

Processors

create_processor loads the processor of a checkpoint, which turns raw inputs into the inputs of its model and its outputs back into results. Each processor type ports a class of the Python library of its checkpoints, with its settings as fields.

Metalhead.create_processor — Function
create_processor(name; revision = nothing, subfolder = "", kwargs...)

The processor of a checkpoint: it turns raw inputs (images, text, boxes) into the inputs of the model, and the model's outputs back into results (boxes, masks, text), as the checkpoint's configuration and its Python library do. create_model makes the forward pass match PyTorch; the processor makes the whole path from raw input to result match.

name is a Hub repository, a local directory or a bare architecture, as for create_model, and validated repositories default to the commit they were validated at. A branch is read at the commit of the first file found, so that the files come from one commit; offline or without network, from the cache. The settings are read from the repository's files over the defaults of the Python class, strictly: a setting that the port doesn't implement is an error, and an unknown key a warning. The keyword arguments override them by name, e.g. size = (; height = 512), where a NamedTuple merges into a nested setting; each goes to the one component that has a setting of that name, and a name that no component has is an error.

Returns the most complete processor the repository describes, as transformers' AutoProcessor does: the class that its files or its model type name, such as a processor that combines an image processor and a tokenizer (Transformers.Sam3Processor); else its tokenizer, as a TextProcessor; else its image processor, such as the Timm.EvalTransform of every timm checkpoint. A diffusers pipeline and a sentence-transformers model prepare their inputs themselves, and have none. Calling a processor returns what the model takes, a NamedTuple (an array for Timm.EvalTransform), with token indices that start at 1:

using HuggingFaceTokenizers
model = create_model("unsloth/sam3.1")
proc = create_processor("unsloth/sam3.1")
img = rand(UInt8, 640, 480, 3)                       # width × height × channels
inputs = proc(; images = img, text = "a cat")
outputs = model(inputs)
post_process_instance_segmentation(proc, outputs; target_sizes = inputs.original_sizes)

Images are arrays of size (width, height, channels), of integers from 0 to 255 or of reals from 0 to 1, or, with ImageCore loaded, matrices of colorants (see image_array). The processor classes are ported along with their models, and a class that isn't ported is an error. show prints each setting and where it comes from.

Metalhead ports no tokenizer: the tokenizer of a repository is HuggingFaceTokenizers.jl's Tokenizer of its tokenizer.json, which needs using HuggingFaceTokenizers before create_processor. Its token ids start at 0, the token indices of the models and of processors at 1, and decode(proc, indices) decodes token indices:

using HuggingFaceTokenizers
model = create_model("HuggingFaceTB/SmolLM2-135M")
tok = create_processor("HuggingFaceTB/SmolLM2-135M")
logits = model(tok("The capital of France is"))
decode(tok, argmax(logits[:, end]))   # " Paris"
source
Metalhead.TextProcessor — Type
TextProcessor(tokenizer; model_input_names = ["input_ids", "attention_mask"])

The processor of a repository whose processor is its tokenizer, such as that of a language model or a text encoder, where transformers' AutoProcessor returns the tokenizer: HuggingFaceTokenizers.jl's Tokenizer of the repository's tokenizer.json, which create_processor loads after using HuggingFaceTokenizers, with the padding token of its tokenizerconfig.json. Its one setting, `modelinputnames, is that of transformers' tokenizer: the keys that it returns, in order, from"inputids","tokentypeids"and"attentionmask". It is themodelinputnames` of tokenizerconfig.json, else that of the tokenizer_class there (BertTokenizer returns token_type_ids too), else ["input_ids", "attention_mask"].

(processor::TextProcessor)(text; text_pair = nothing, add_special_tokens = true)
(processor::TextProcessor)(; text, text_pair = nothing, add_special_tokens = true)

The inputs of a language model or text encoder from text, a string or a vector of strings, which the tokenizer encodes, with the special tokens that it adds around them unless add_special_tokens = false, as a NamedTuple that the model takes as it is (model(inputs)). text_pair, of the same form and size, encodes each text with a second one, as transformers' tokenizer(text, text_pair) does with the template of the tokenizer ([CLS] a [SEP] b [SEP] for BERT, <s> a </s></s> b </s> for XLM-RoBERTa): the input of a cross-encoder, such as a reranker, for a query and a passage.

KeySizeContents
input_ids(seq_len, batch)token indices of the texts, 1-based, padded on the right
token_type_ids(seq_len, batch)0 for the tokens of the text, 1 for those of its pair; 0 for the padding
attention_mask(seq_len, batch)1 for the tokens, 0 for the padding

One text gives vectors (seq_len,), so that the logits of its next token are model(inputs)[:, end]. A batch of texts is padded on the right to the longest, with the padding token of the tokenizer, as transformers pads it with padding = true; a tokenizer without a padding token pads with the token index 1, which the attention_mask keeps out of the attention, so that the outputs of each text are those it has alone. The texts are not truncated, as transformers' tokenizers by default. token_type_ids are the segment ids of transformers, 0 and 1, not token indices.

Token indices start at 1, the token ids of the tokenizer at 0. decode(processor, indices) (HuggingFaceTokenizers' decode, which takes the keywords of the tokenizer's, such as skip_special_tokens) decodes token indices, and processor.tokenizer is the tokenizer, which takes and returns token ids.

using Metalhead, HuggingFaceTokenizers
model = create_model("HuggingFaceTB/SmolLM2-135M")
tok = create_processor("HuggingFaceTB/SmolLM2-135M")
logits = model(tok("The capital of France is"))
decode(tok, argmax(logits[:, end]))   # " Paris"

reranker = create_model("BAAI/bge-reranker-base")
proc = create_processor("BAAI/bge-reranker-base")
reranker(proc(["what is a panda?", "what is a panda?"];
    text_pair = ["The giant panda is a bear.", "Paris is in France."]))   # 1 × 2 scores
source
Metalhead.Timm.EvalTransform — Type
Timm.EvalTransform(; input_size = (3, 224, 224), interpolation = "bicubic",
                   mean = (0.485, 0.456, 0.406), std = (0.229, 0.224, 0.225),
                   crop_pct = 0.875, crop_mode = "center")

timm's evaluation transform, create_transform(**resolve_data_config(...)): the image processor of every timm checkpoint, which create_processor builds from the pretrained_cfg of its config.json, resolved as timm's resolve_data_config resolves it: a setting that the configuration doesn't have, or that is false in Python ("", [], 0), takes the default above.

Called on one image or a vector of images, it returns the input of the model, not a NamedTuple: a Float32 array of size (W, H, 3, batch), where (3, H, W) is input_size, of the images resized, cropped and normalized, bit for bit as timm transforms them as PIL images. One image gives a batch of one.

model = create_model("timm/convnext_tiny.fb_in1k")
transform = create_processor("timm/convnext_tiny.fb_in1k")
logits = model(transform(image))    # transform(image): 224 × 224 × 3 × 1

An image is an array of size (width, height, channels) (torch's (channels, height, width) reversed) of integers from 0 to 255 (an 8-bit image, as UInt8) or of reals from 0 to 1, which are converted to 8 bits as torchvision's ToPILImage converts them, or, with ImageCore loaded, a matrix of colorants (see image_array). Grey images, and the alpha of RGBA, are converted to RGB as Pillow's convert("RGB") does. Each image is

  1. resized with Pillow's filter interpolation ("nearest", "bilinear", "bicubic", "box", "hamming" or "lanczos"): its short side to floor(input_size / crop_pct) with crop_mode = "center", both sides with crop_mode = "squash". (A non-square input_size with "center" resizes with "bilinear", as in timm.) timm's third crop mode, "border", is not supported;
  2. center-cropped to the height and width of input_size (channels, height, width), padded with zeros where it is smaller;
  3. scaled to [0, 1] and normalized with mean and std.

The settings also take the forms that timm's create_transform takes: input_size as (height, width) or an integer, the side of a square; mean and std as one value for all channels; interpolation and crop_mode as symbols; and crop_pct and crop_mode as nothing for their defaults.

create_processor(repo; use_test_size = true) resolves test_input_size and test_crop_pct in place of input_size and crop_pct, where the configuration has them, as resolve_data_config(use_test_size = True) does: many checkpoints are evaluated at a larger size than the one they were trained at. The other keywords of create_processor override the settings, e.g. crop_pct = 1.0. Timm.pretrained_cfg returns the pretrained_cfg as it is.

source
Metalhead.Transformers.Qwen3VLProcessor — Type
Qwen3VLProcessor(image_processor, tokenizer)

transformers' processor of Qwen3-VL and Qwen3.5: a Qwen2VLImageProcessor and the tokenizer of the repository, HuggingFaceTokenizers.jl's Tokenizer, which create_processor loads from a repository such as "Qwen/Qwen3.5-0.8B". The tokenizer needs using HuggingFaceTokenizers before create_processor; without it, tokenizer is nothing, and the processor takes images but no text.

(processor::Qwen3VLProcessor)(; images = nothing, text = nothing)

The inputs of a Qwen3.5 vision-language model (qwen3_5_for_conditional_generation) from images (one image or a vector of them, see Qwen2VLImageProcessor) and text, as a NamedTuple with the keys of transformers' Qwen3VLProcessor, in Julia's layout (PyTorch's dimensions reversed), which the model takes as it is (model(inputs)): the outputs of the tokenizer and of the image processor, merged.

KeySizeContents
input_ids(seq_len, batch)token indices of the texts, 1-based, padded on the right, with the image placeholders expanded; with text
attention_mask(seq_len, batch)1 for the tokens, 0 for the padding; with text
mm_token_type_ids(seq_len, batch)1 for image tokens, 2 for video tokens, 0 for the others; with text
pixel_values(D, num_patches)the patches of the images, resized and normalised, Float32; with images
image_grid_thw(3, num_images)the numbers of patches (1, height ÷ patch_size, width ÷ patch_size) of each image; with images

where D = 3 * temporal_patch_size * patch_size^2 (1536 for Qwen3.5).

text is a string or a vector of strings, which the tokenizer encodes with its special tokens. One text gives vectors (seq_len,), as the model takes them, so that the logits of its next token are model(inputs)[:, end]. A batch of texts is padded on the right to the longest, with the padding token of the tokenizer, as transformers pads it with padding = true; the other keywords of transformers' tokenizer (truncation, max_length) aren't ported.

Each image placeholder of the texts, <|image_pad|>, stands for the next image, in order through the texts, and becomes as many image tokens as the image has merged patches, prod(image_grid_thw[:, i]) ÷ merge_size^2, as in transformers. The numbers of placeholders and of images must be equal (transformers fails on more placeholders than images, and its model on fewer). The chat template isn't applied: write the placeholders where it would, <|vision_start|><|image_pad|><|vision_end|>.

using Metalhead, HuggingFaceTokenizers
model = create_model("Qwen/Qwen3.5-0.8B")
proc = create_processor("Qwen/Qwen3.5-0.8B")
image = rand(UInt8, 640, 480, 3)                       # width × height × channels
prompt = "<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>" *
         "Describe this image.<|im_end|>\n<|im_start|>assistant\n" *
         "<think>\n\n</think>\n\n"
inputs = proc(; images = image, text = prompt)        # 300 image tokens for 30 × 40 patches
logits = model(inputs)                                 # vocab_size × seq_len
HuggingFaceTokenizers.decode(proc.tokenizer, [argmax(logits[:, end]) - 1])
source
Metalhead.Transformers.Qwen2VLImageProcessor — Type
Qwen2VLImageProcessor(; do_resize = true, resample = 3,
                      size = (; shortest_edge = 3136, longest_edge = 1003520),
                      do_rescale = true, rescale_factor = 1 / 255, do_normalize = true,
                      image_mean = (0.48145466, 0.4578275, 0.40821073),
                      image_std = (0.26862954, 0.26130258, 0.27577711),
                      do_convert_rgb = true, patch_size = 14, temporal_patch_size = 2,
                      merge_size = 2)

transformers' image processor of Qwen2-VL, with the defaults of its Python class; the image processor of a Qwen3VLProcessor, with the settings of a repository such as "Qwen/Qwen3.5-0.8B" (patches of 16 pixels, mean and standard deviation 0.5). The keys min_pixels and max_pixels of Qwen2-VL's configurations, or of create_processor, set size.shortest_edge and size.longest_edge, as in transformers. Called on one image or a vector of images, it returns the inputs of the vision encoder of Qwen3.5 (qwen3_5_vision_model), which takes them as they are (model(ip(images))), as a NamedTuple in Julia's layout (PyTorch's dimensions reversed):

KeySizeContents
pixel_values(D, num_patches)the patches of the images, resized and normalised, Float32
image_grid_thw(3, num_images)the numbers of patches (1, height ÷ patch_size, width ÷ patch_size) of each image

where D = 3 * temporal_patch_size * patch_size^2 (1536 for Qwen3.5). The columns of pixel_values (the rows of transformers') are the patches of the images one after the other, each image ordered by windows of merge_size × merge_size patches, row-major over the windows and within each window. A patch is flattened as an array of size (patch_size, patch_size, temporal_patch_size, 3) (width first), its pixels repeated temporal_patch_size times: an image is a video of one frame.

An image is an array of size (width, height, channels) (torch's (channels, height, width) reversed) of integers from 0 to 255 (an 8-bit image, as UInt8) or of reals from 0 to 1. Each image is

  1. converted to RGB with do_convert_rgb: a grey image (a matrix, or one channel, or two with alpha) has its grey repeated, and the alpha of RGBA is dropped, as PIL does;
  2. resized with do_resize, with PIL's filter resample, 2 (bilinear) or 3 (bicubic), antialiased as torchvision resizes it, to the size whose sides are multiples of patch_size * merge_size, with an area from size.shortest_edge to size.longest_edge pixels, that keeps the aspect ratio of the image best (transformers' smart_resize). Without do_resize, the sides of an image must be multiples of patch_size * merge_size;
  3. multiplied by rescale_factor with do_rescale, and normalized with image_mean and image_std with do_normalize. An image of reals from 0 to 1 is taken as an 8-bit image already rescaled: do_rescale leaves it as it is, and without it the values are multiplied by 255;
  4. cut into patches of patch_size × patch_size pixels.

An 8-bit image gives the same pixel_values as transformers, and an image of reals the same to Float32 precision. The settings are checked as the processor is made, and also take the forms that transformers takes: resample as the name of PIL's filter (:bicubic); image_mean and image_std as one value for all channels.

source

Post-processing takes the processor and the outputs of the model:

Metalhead.post_process_object_detection — Function
post_process_object_detection(processor, outputs; kwargs...)

The objects that a model detects in each image, from its outputs and its processor (transformers' post_process_object_detection): for each image, a NamedTuple of their scores and boxes. The processor of each model has a method; that of SAM 3 is Sam3Processor.

source
Metalhead.post_process_instance_segmentation — Function
post_process_instance_segmentation(processor, outputs; kwargs...)

The objects that a model segments in each image, from its outputs and its processor (transformers' post_process_instance_segmentation): for each image, a NamedTuple of their scores, boxes and masks. The processor of each model has a method; that of SAM 3 is Sam3Processor.

source
Metalhead.post_process_semantic_segmentation — Function
post_process_semantic_segmentation(processor, outputs; kwargs...)

The semantic segmentation of each image by a model, from its outputs and its processor (transformers' post_process_semantic_segmentation). The processor of each model has a method; that of SAM 3 is Sam3Processor.

source

Image processors take images in one layout (and so do pipelines, their reference images), to which an extension for another type of image adds a method, and image_matrix turns arrays of that layout, such as generated images, back into images of colorants:

Metalhead.image_array — Function
image_array(image) -> AbstractArray{<:Real, 3}

image as processors take it: an array of size (width, height, channels) (torch's (channels, height, width) reversed) of integers from 0 to 255 (an 8-bit image, such as UInt8) or of reals from 0 to 1. A matrix is a grey image of one channel.

With ImageCore loaded (as by Images.jl), it also takes a matrix of colorants of size (height, width), such as the Matrix{RGB{N0f8}} that FileIO.load returns, as permutedims(channelview(img), (3, 2, 1)) but with the channels of any colorant in the order R, G, B (then alpha), or gray (then alpha). N0f8 values, those of 8-bit images, become the UInt8 values they store, and others reals from 0 to 1.

source
Metalhead.image_matrix — Function
image_matrix(x) -> AbstractMatrix{<:Colorant}

The inverse of image_array: the array x of size (width, height, channels), such as an image that a pipeline generates or a mask of SAM 3, as a matrix of colorants of size (height, width), which FileIO saves and image viewers show. It needs ImageCore (or Images.jl) loaded.

The channels make RGB (3), RGBA (4), gray (1) or gray and alpha (2) colorants; a matrix (width, height), such as a mask, a gray image. 8-bit values (integers from 0 to 255) make N0f8 colorants, Bools gray Bools, and other reals, from 0 to 1, Float32 colorants. An array of size (width, height, channels, batch) gives a vector of images.

using ImageCore, FileIO
image = pipe("A photo of a red fox sitting in fresh snow"; num_inference_steps = 4)
FileIO.save("fox.png", Metalhead.image_matrix(cpu(image))[1])
source

timm

Metalhead.Timm.pretrained_cfg — Function
Metalhead.Timm.pretrained_cfg(name; revision = nothing) -> NamedTuple

Preprocessing settings of a timm checkpoint (mean, std, input_size, crop_pct, interpolation, ...), as stored in its config.json. name is a Hub repository, a bare architecture (for its default checkpoint) or a local directory, as for Metalhead.create_model. input_size is in PyTorch order (C, H, W). Metalhead.create_processor returns the transform they describe, a Timm.EvalTransform.

source

transformers

Metalhead.Transformers — Module
Metalhead.Transformers

Flux ports of Hugging Face Transformers models. Each ported model mirrors the PyTorch module tree, so that pretrained state_dicts load by name; build them with Metalhead.create_model from a Hub repository.

Language models take token indices starting at 1, as Flux's Embedding does: add 1 to the ids of a Hugging Face tokenizer, and subtract 1 from the index of a logit to get its token id. With using HuggingFaceTokenizers, Metalhead.create_processor loads the tokenizer of a repository.

source
Metalhead.Transformers.KVCache — Type
KVCache(model, batch_size, max_len)

The key-value cache of a causal language model for batch_size sequences of up to max_len tokens each. With it, each step of a generation loop runs the model on the new tokens only, rather than on the whole sequence so far. It holds the keys and values of the tokens seen, for each layer, in buffers allocated at their full length, with the type and on the device of the model's weights, and lengths, a Vector{Int} on the host with the number of tokens of each sequence. The keys are stored rotated.

Llama, Qwen2, Qwen3, Qwen3 MoE and the Granite models (their attention layers are all full attention) have a cache; Gemma 3 (sliding-window attention) and Qwen3.5 (a gated delta rule) raise a "not supported yet" error.

model(input_ids; cache) runs the model on the token indices input_ids of size (n, batch_size), which follow the tokens in the cache, stores their keys and values and returns (logits, cache), the cache being updated in place; forward_features(model, input_ids; cache) returns (features, cache) the same way. A vector of token indices is a batch of one. The rotary positions of the new tokens continue those of their sequence.

  • The first call, with an empty cache, is the prefill of the prompts: a batch of sequences of different lengths is padded on the right, and its attention_mask (see llama_for_causal_lm) sets the lengths. It computes what the model computes without a cache. The logits of the next token of a sequence are at its last token, logits[:, lengths[b], b], which is not the last position of a shorter one: last_tokens picks them.
  • The next calls are steps of n ≥ 1 new tokens (the same number for all the sequences, without attention_mask), written after the tokens of each sequence, whatever the lengths of the others. A step of a few tokens, such as the chunk of a long prompt, attends to the cache and, causally, to the tokens of the chunk. Their logits are those of the same tokens in a forward pass of the whole sequence, up to rounding.

The sequences of a batch of the same length write their keys with one copy; otherwise, each sequence writes its own. max_len bounds the first call too, padding included. The cache is for inference: gradients do not flow through it. empty!(cache) makes it ready for new sequences.

Examples

A greedy loop over a batch of two texts of different lengths (tok is a TextProcessor):

using Metalhead.Transformers: KVCache, last_tokens

inputs = tok(["The capital of France is", "Hello"])           # padded on the right
cache = KVCache(model, size(inputs.input_ids, 2), 64)
logits, _ = model(inputs.input_ids; inputs.attention_mask, cache)    # prefill
next = argmax.(eachcol(last_tokens(logits, cache.lengths)))     # one token per sequence
for _ in 1:8
    logits, _ = model(reshape(next, 1, :); cache)               # a step of one token
    next = argmax.(eachcol(dropdims(logits; dims = 2)))
end
source
Metalhead.Transformers.last_tokens — Function
last_tokens(x, lengths)

The columns of x, of size (features, seq_len, batch_size), at the position lengths[b] of each sequence b: an array of size (features, batch_size). For the logits of a KVCache prefill, with cache.lengths as lengths, these are the logits of the next token of each sequence of a batch padded on the right, whose last position is padding for the shorter ones.

source
Metalhead.Transformers.get_text_features — Function
get_text_features(model::CLIPModel, input_ids; attention_mask = nothing)
get_text_features(model::CLIPModel, inputs::NamedTuple)

The output of the text encoder of CLIP on the token indices input_ids (seqlen, numtexts), 1-based and padded on the right after the ones of attention_mask, or on the input_ids (and attention_mask) of the output of its processor: its last_hidden_state and, as pooler_output (projectiondim, numtexts), its pooled features projected by text_projection, not normalised (transformers' get_text_features).

source
get_text_features(model::Sam3Model, input_ids; attention_mask = nothing)

The features of the text prompts of SAM 3: the last_hidden_state of its text encoder on the token indices input_ids (seqlen, batchsize), 1-based and padded on the right after the ones of attention_mask, and its projection to the hidden size of the detector (pooler_output, as transformers names it). Pass them to the model in place of the token indices to reuse them for several images.

source
get_text_features(model::Union{SiglipModel, Siglip2Model}, input_ids;
                  attention_mask = nothing)
get_text_features(model::Union{SiglipModel, Siglip2Model}, inputs::NamedTuple)

The output of the text model of SigLIP or SigLIP 2 on the token indices input_ids (seqlen, numtexts), 1-based, or on the input_ids (and attention_mask) of the output of its processor: (; last_hidden_state, pooler_output) (see SiglipTextModel), whose pooler_output (projectionsize, numtexts) are the embeddings of the texts before their normalization, as transformers' get_text_features returns them.

source
Metalhead.Transformers.get_image_features — Function
get_image_features(model::CLIPModel, pixel_values; interpolate_pos_encoding = false)
get_image_features(model::CLIPModel, inputs::NamedTuple;
                   interpolate_pos_encoding = false)

The output of the vision encoder of CLIP on the images pixel_values (width, height, channels, numimages), or on the `pixelvaluesof the output of its processor: itslasthiddenstateand, aspooleroutput` (projectiondim, numimages), its pooled features projected by `visualprojection, not normalised (transformers'getimagefeatures). See [CLIPVisionModel](@ref Metalhead.Transformers.CLIPVisionModel) forinterpolateposencoding`.

source
get_image_features(model::SiglipModel, pixel_values; interpolate_pos_encoding = false)
get_image_features(model::SiglipModel, inputs::NamedTuple)

The output of the vision model of SigLIP on the images pixel_values (width, height, channels, numimages), or on the `pixelvaluesof the output of its processor:(; lasthiddenstate, pooleroutput)(see [SiglipVisionModel](@ref Metalhead.Transformers.SiglipVisionModel)), whosepooleroutput(hidden, num_images) are the embeddings of the images before their normalization, as transformers'getimagefeatures` returns them.

source
get_image_features(model::Siglip2Model, pixel_values; spatial_shapes,
                   pixel_attention_mask = nothing)
get_image_features(model::Siglip2Model, inputs::NamedTuple)

The output of the vision model of SigLIP 2 on images given as patches (see Siglip2VisionModel), or on the output of its image processor: (; last_hidden_state, pooler_output), whose pooler_output (hidden, numimages) are the embeddings of the images before their normalization, as transformers' `getimage_features` returns them.

source
Metalhead.Transformers.Qwen3_5ForConditionalGeneration — Type
(model::Qwen3_5ForConditionalGeneration)(inputs::NamedTuple)
(model::Qwen3_5Model)(inputs::NamedTuple)

The model on the output of a Qwen3VLProcessor: its input_ids and attention_mask, and pixel_values and image_grid_thw if it has images; other keys (mm_token_type_ids) are ignored. The input_ids of one text, a vector, give the logits (or hidden states) of size (vocab_size, seq_len), so that logits[:, end] are those of the next token. A batch is padded on the right, as the processor pads it, and each text gets the outputs it gets alone, at the positions of its tokens.

source
Metalhead.Transformers.Qwen3_5VisionModel — Type
(model::Qwen3_5VisionModel)(pixel_values, grid_thw) -> (; last_hidden_state, pooler_output)

Encode images and videos, given as patches: pixel_values of size (in_channels * temporal_patch_size * patch_size^2, num_patches), and grid_thw, an integer matrix of size (3, num_images) whose column (t, h, w) gives the numbers of patches of an image along time, height and width (see qwen3_5_vision_model). Returns the hidden states of the patches, of size (hidden_size, num_patches), and the merged tokens of the pooler_output, of size (out_hidden_size, num_patches ÷ spatial_merge_size^2), image after image.

source
(model::Qwen3_5VisionModel)(inputs::NamedTuple)

The model on the output of a Qwen2VLImageProcessor (or of a Qwen3VLProcessor with images): its pixel_values and image_grid_thw; other keys are ignored.

source

The models and processors of CLIP, of the text encoders BERT, XLM-RoBERTa and ModernBERT, of SAM 3, and of SigLIP and SigLIP 2, have their own API references: CLIP, BERT, XLM-RoBERTa and ModernBERT, SAM 3, and SigLIP and SigLIP 2.

diffusers

Metalhead.Diffusers — Module
Metalhead.Diffusers

Flux ports of Hugging Face Diffusers models and pipelines. Each ported model mirrors the PyTorch module tree, so that pretrained state_dicts load by name. Metalhead.create_model builds a whole pipeline from a repository, such as Flux2KleinPipeline from "black-forest-labs/FLUX.2-klein-4B" or ZImagePipeline from "Tongyi-MAI/Z-Image-Turbo", or one of its components with subfolder.

Images are WHCN arrays with values in [-1, 1] for the autoencoders, and [0, 1] for the output of pipelines. The reference images of a pipeline are images as processors take them (see Metalhead.image_array).

source
Metalhead.Diffusers.Flux2KleinPipeline — Type
Flux2KleinPipeline

diffusers' pipeline of FLUX.2 [klein], which generates images from text, or edits images (reference images, image) as the text says, built by create_model from a repository such as "black-forest-labs/FLUX.2-klein-4B". Its components are text_encoder, a Qwen3 language model whose hidden states condition the image; transformer, a Flux2Transformer2DModel that predicts the flow of the latents; vae, the AutoencoderKLFlux2 that decodes them (and encodes the reference images); scheduler, the FlowMatchEulerDiscreteScheduler; and tokenizer, the tokenizer of the text encoder: HuggingFaceTokenizers.jl's Tokenizer of the tokenizer/ subfolder. Without using HuggingFaceTokenizers before create_model, it is a reference to the tokenizer.json, which the pipeline loads once HuggingFaceTokenizers is loaded, without loading the pipeline again; it is nothing for a pipeline saved without a tokenizer, which takes prompt embeddings only. is_distilled checkpoints (FLUX.2 [klein] 4B, 9B) sample in a few steps without classifier-free guidance; the base ones (FLUX.2 [klein] base 4B, 9B) take about 50 steps and a guidance_scale of 4.

gpu(pipeline) and bf16(pipeline) move and convert every component but the tokenizer.

using Metalhead, Flux, HuggingFaceTokenizers
using CUDA, cuDNN

pipe = create_model("black-forest-labs/FLUX.2-klein-4B"; eltype = BFloat16)
pipe = gpu(pipe)

image = pipe("A cat holding a sign that says hello world"; num_inference_steps = 4)
edited = pipe("Make the sign say goodbye"; image = image[:, :, :, 1], num_inference_steps = 4)
source
Metalhead.Diffusers.encode_prompt — Function
encode_prompt(pipeline, prompt; max_sequence_length = 512,
              text_encoder_out_layers = (9, 18, 27))
encode_prompt(pipeline, input_ids; attention_mask = nothing,
              text_encoder_out_layers = (9, 18, 27))

The embeddings of a prompt (a string) or of a vector of prompts, as diffusers' encode_prompt makes them, of size (length(text_encoder_out_layers) * hidden_size, max_sequence_length, batch). Each prompt is formatted with the chat template of the Qwen3 text encoder (a user message, with thinking disabled and a generation prompt), tokenized with pipeline.tokenizer (which needs using HuggingFaceTokenizers), truncated to max_sequence_length tokens and padded on the right to them. The embeddings are the hidden states of the layers text_encoder_out_layers of the text encoder (transformers' output_hidden_states, of which that of the last layer is normed), concatenated along the features.

The second method takes the prompts tokenized: input_ids, 1-based token indices of size (seq_len, batch), padded on the right as attention_mask says, and moved to the device of the text encoder.

source
encode_prompt(pipeline::ZImagePipeline, prompt; max_sequence_length = 512)
encode_prompt(pipeline::ZImagePipeline, input_ids; attention_mask = nothing)

The embeddings of a prompt (a string) or of a vector of prompts, as diffusers' ZImagePipeline.encode_prompt makes them: a vector with a matrix of size (hidden_size, length) for each prompt, the hidden states of its tokens in the second-to-last layer of the text encoder (transformers' output_hidden_states[-2]), without the padding. Each prompt is formatted with the chat template of the Qwen3 text encoder (a user message and a generation prompt, with thinking enabled), tokenized with pipeline.tokenizer (which needs using HuggingFaceTokenizers) and truncated to max_sequence_length tokens.

The second method takes the prompts tokenized: input_ids, 1-based token indices of size (seq_len, batch), padded on the right as attention_mask says.

source
Metalhead.Diffusers.ZImagePipeline — Type
ZImagePipeline

diffusers' text-to-image pipeline of Z-Image, built by create_model from a repository such as "Tongyi-MAI/Z-Image-Turbo". Its components are text_encoder, a Qwen3 decoder (a LlamaModel, transformers' Qwen3Model) whose hidden states condition the image; transformer, a ZImageTransformer2DModel that predicts the flow of the latents; vae, the AutoencoderKL that decodes them; scheduler, the FlowMatchEulerDiscreteScheduler; and tokenizer, the tokenizer of the text encoder, as for Flux2KleinPipeline: HuggingFaceTokenizers.jl's Tokenizer of the tokenizer/ subfolder, a reference to it that the pipeline loads once HuggingFaceTokenizers is loaded, or nothing without one.

The defaults of the pipeline are those of diffusers, for Z-Image: 50 steps with a guidance_scale of 5. Z-Image-Turbo is distilled to sample in 9 steps without guidance (guidance_scale = 0).

gpu(pipeline) and bf16(pipeline) move and convert every component but the tokenizer.

using Metalhead, Flux, HuggingFaceTokenizers
pipe = create_model("Tongyi-MAI/Z-Image-Turbo"; eltype = BFloat16)
image = pipe("A cat holding a sign that says hello world"; num_inference_steps = 9,
             guidance_scale = 0)
source

sentence-transformers

create_model builds a SentenceTransformer from a repository with a modules.json. Its classes and functions have their own API reference.

Internals

The Internals page documents what is not public: the builders, the components of the pipelines and the helpers of the ported models, which may change without notice. The Porting page documents how a library registers its architectures, also internal.