Internals

convmixer(; dim, depth, kernel_size = 9, patch_size = 7, num_classes = 1000, kwargs...)

Port of timm's ConvMixer. Keyword arguments are those of the timm class, with act_layer given by name ("gelu" or "relu"); kernel_size and patch_size are integers or (height, width) tuples.

convnext(; depths, dims, num_classes = 1000, in_chans = 3, kwargs...)

Port of timm's ConvNeXt with the patch stem and the default head (ConvNeXt, ConvNeXt-Atto to Nano). Keyword arguments are those of the timm class.

densenet(; growth_rate = 32, block_config = (6, 12, 24, 16), kwargs...)

Port of timm's DenseNet (DenseNet-BC, with the optional deep stem and blur pooling). Keyword arguments are those of the timm class, except that aa_layer is "blurpool2d" or nothing. memory_efficient is accepted and ignored: it only changes how gradients are computed.

efficientnet(; block_args, num_classes = 1000, num_features = 1280, kwargs...)

Port of timm's EfficientNet class (EfficientNet, EfficientNet-V2, MobileNet-V1/V2, MNASNet and the others built by timm.models.efficientnet). block_args comes from decode_arch_def; norm_layer is nothing or the arguments of BatchNorm2d, and se_layer(channels; act_layer, rd_ratio) creates the squeeze-and-excitation modules. Other keyword arguments are those of the timm class.

inception_resnet_v2(; num_classes = 1000, in_chans = 3, drop_rate = 0.0)

Port of timm's InceptionResnetV2. Keyword arguments are those of the timm class.

inception_v3(; num_classes = 1000, in_chans = 3, drop_rate = 0.0)

Port of timm's InceptionV3 without the auxiliary classifier (aux_logits = false). Keyword arguments are those of the timm class. The input must be at least 75x75.

inception_v4(; num_classes = 1000, in_chans = 3, drop_rate = 0.0)

Port of timm's InceptionV4. Keyword arguments are those of the timm class.

mlp_mixer(; img_size = 224, patch_size = 16, num_blocks = 8, embed_dim = 512, kwargs...)

Port of timm's MlpMixer: MLP-Mixer, ResMLP, gMLP and the GLU mixer. Keyword arguments are those of the timm class, with the classes it takes given by name: block_layer is "MixerBlock", "ResBlock" or "SpatialGatingBlock", mlp_layer is "Mlp", "GluMlp" or "GatedMlp" (only the pairs of timm's entry points), norm_layer is "layernorm" (eps = 1e-6) or "affine" and act_layer is "gelu" or "silu". init_values is the layer scale of ResBlock (functools.partial(ResBlock, init_values = ...) in timm). global_pool is "avg" or "", which applies the head to every token.

mobilenetv3(; block_args, num_classes = 1000, num_features = 1280, kwargs...)

Port of timm's MobileNetV3 class (MobileNet-V3, LCNet, MobileNet-V4): the EfficientNet stem and blocks with the "efficient head", where global pooling comes before the head convolution and there is no norm layer before the classifier (but one after the head convolution with head_norm). num_features is the width of the head convolution. block_args comes from decode_arch_def, norm_layer is nothing or the arguments of BatchNorm2d, and se_layer(channels; act_layer, rd_ratio) creates the squeeze-and-excitation modules. Other keyword arguments are those of the timm class.

resnet(; block, layers, num_classes = 1000, in_chans = 3, kwargs...)

Port of timm's ResNet (ResNet, ResNeXt, Wide ResNet, SE-ResNet, ECA-ResNet, Res2Net and the c/d/s/t stem variants). Keyword arguments are those of the timm class. block_args is a NamedTuple of keywords for the blocks: attn_layer = "se" or "eca" for BasicBlock and Bottleneck, scale for Bottle2neck.

vgg(; cfg, num_classes = 1000, in_chans = 3, norm_layer = nothing, kwargs...)

Port of timm's VGG. cfg lists the output channels of the 3x3 convolutions, with "M" for a max pool, and norm_layer = "batchnorm2d" adds batch normalisation (the _bn variants). Other keyword arguments are those of the timm class.

vit(; img_size = 224, patch_size = 16, embed_dim = 768, depth = 12, num_heads = 12, kwargs...)

Port of timm's VisionTransformer, including the options of DINOv2 (init_values, reg_tokens, no_embed_class), CLIP (pre_norm, act_layer = "quick_gelu") and SigLIP (class_token = false, global_pool = "map"), and dynamic_img_size. Keyword arguments are those of the timm class, except that the epsilon of norm_layer is given as norm_eps (timm's partial(LayerNorm, eps = ...)). act_layer is "gelu" (the default), "gelu_tanh" or "quick_gelu". img_size and patch_size are integers or (height, width) tuples, as in timm. Other options (qk_norm, custom layers, ...) error.

xception_aligned(; block_cfg, num_classes = 1000, in_chans = 3, kwargs...)

Port of timm's XceptionAligned (Aligned Xception, with the pre-activation variant). Keyword arguments are those of the timm class, except that norm_eps stands for the eps of a norm_layer = partial(nn.BatchNorm2d, eps = ...). block_cfg is a sequence of named tuples (in_chs, out_chs, stride) with the optional no_skip and start_with_relu.

gemma3_for_causal_lm(; vocab_size = 262208, hidden_size = 2304, num_hidden_layers = 26, kwargs...)

Port of transformers' Gemma3ForCausalLM, the Gemma 3 text model, built from the fields of its Gemma3TextConfig with the same names and defaults, and rope_theta, rope_local_base_freq, rope_scaling and sliding_window_pattern, which older config.jsons have in place of rope_parameters and layer_types. It takes and returns the arrays of llama_for_causal_lm.

Gemma 3 differs from Llama in its decoder layers: the token embeddings are multiplied by √hidden_size, the RMSNorms scale by 1 + weight, the outputs of the attention and of the MLP are normalised too, the queries and keys are normalised per head and scaled by query_pre_attn_scalar^-0.5, the MLP gate is gelu_pytorch_tanh, and the logits may be softcapped (final_logit_softcapping). The layers attend either to all the previous tokens ("full_attention") or to the last sliding_window of them ("sliding_attention"), as layer_types says, with rotary embeddings of their own: rope_parameters has one for each layer type. Bidirectional attention (use_bidirectional_attention) errors. attn_logit_softcapping is ignored, as transformers ignores it.

granite_for_causal_lm(; vocab_size = 32000, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' GraniteForCausalLM (IBM Granite 3 and 4.1), built from the fields of its GraniteConfig with the same names and defaults, rope_theta and rope_scaling, which older config.jsons have in place of rope_parameters, and head_dim, which the attention reads where a config.json has it. It takes and returns the arrays of llama_for_causal_lm.

Granite is Llama with four multipliers: the token embeddings are multiplied by embedding_multiplier, the attention scores by attention_multiplier (in place of 1/√head_dim), the outputs of the attention and of the MLP by residual_multiplier, and the logits are divided by logits_scaling.

granitemoe_for_causal_lm(; vocab_size = 32000, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' GraniteMoeForCausalLM (IBM Granite 3 MoE), built from the fields of its GraniteMoeConfig with the same names and defaults, and the keys of granite_for_causal_lm that it adds (rope_theta, rope_scaling, head_dim). It takes and returns the arrays of llama_for_causal_lm.

Granite MoE is Granite whose MLP is a sparse mixture of num_local_experts gated MLPs of intermediate_size features: each token goes through the num_experts_per_tok experts of largest router logit, whose outputs are weighted by the softmax of these logits. The training options (output_router_logits, router_aux_loss_coef) are not supported.

granitemoehybrid_for_causal_lm(; vocab_size = 32000, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' GraniteMoeHybridForCausalLM (IBM Granite 4.0) without its Mamba layers: the models whose layer_types are all "attention" (or "full_attention"), such as granite-4.0-350m and granite-4.0-1b. A layer is that of granitemoeshared_for_causal_lm, whose shared expert is always present, with no mixture of experts if num_local_experts = 0. The attention has a rotary embedding only with position_embedding_type = "rope". Mamba layers ("mamba" or "linear_attention", the default) error, and the mamba_* keys of config.json are ignored.

granitemoeshared_for_causal_lm(; vocab_size = 32000, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' GraniteMoeSharedForCausalLM: Granite MoE (see granitemoe_for_causal_lm) with a shared expert, a gated MLP of shared_intermediate_size features that every token goes through, unless shared_intermediate_size = 0.

llama_for_causal_lm(; vocab_size = 32000, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' LlamaForCausalLM, built from the fields of its LlamaConfig (the config.json of a checkpoint), with the same names and defaults. create_model reads the generation settings of the checkpoint into its generation_config (see GenerationConfig). It ignores the keys of config.json that transformers records or doesn't read, and warns about any other that this function doesn't take. The rotary embedding is the default one or, with rope_scaling or rope_parameters, one with rope_type = "linear" or "llama3" (Llama 3.1 and later); other types error.

The model maps token indices of size (seq_len, batch_size), or a vector for a single sequence, to logits of size (vocab_size, seq_len, batch_size), with causal attention. Token indices start at 1: add 1 to the ids of a Hugging Face tokenizer. A batch of sequences of different lengths, padded on the right, takes the tokenizer's attention_mask of size (seq_len, batch_size) as a keyword argument, which keeps the padding out of the keys. The output of its processor, a TextProcessor, can replace both: model(tok(texts)). forward_features returns the hidden states after the final norm, and forward_intermediates those of each layer.

With a KVCache, model(input_ids; cache) takes the new tokens of each sequence only and returns (logits, cache).

qwen2_for_causal_lm(; vocab_size = 151936, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' Qwen2ForCausalLM, built from the fields of its Qwen2Config with the same names and defaults, and head_dim, which Qwen2Config lacks but transformers reads. transformers derives Qwen2 from Llama's classes, adding a bias to the query, key and value projections, so the model is a LlamaForCausalLM (see llama_for_causal_lm for its inputs and outputs). Sliding-window attention (use_sliding_window, layer_types) and rotary embeddings other than the default, "linear" and "llama3" ones error.

qwen3_5_for_causal_lm(; vocab_size = 248320, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' Qwen3_5ForCausalLM, the text model of Qwen3.5: the decoder of qwen3_5_text_model, with its keywords, and the output projection lm_head, tied to the token embeddings with tie_word_embeddings. It takes and returns the arrays of llama_for_causal_lm, and the keyword position_ids of qwen3_5_decode. The multi-token prediction weights (mtp.*) of the checkpoints are not loaded, as in transformers.

qwen3_5_for_conditional_generation(; text_config, vision_config, image_token_id = 248056,
                                   tie_word_embeddings = false)

Port of transformers' Qwen3_5ForConditionalGeneration, the vision-language model of the Qwen3.5 checkpoints, built from its Qwen3_5Config: the vision encoder qwen3_5_vision_model of vision_config, the decoder qwen3_5_text_model of text_config (both NamedTuples of their config's fields; other keys are ignored) and the output projection lm_head, tied to the token embeddings with tie_word_embeddings. Videos and the multi-token prediction weights (mtp.*) of the checkpoints are not supported.

The model maps token indices of size (seq_len, batch_size), or a vector, to logits of size (vocab_size, seq_len, batch_size), as llama_for_causal_lm does, and takes the keyword arguments attention_mask and, for images, pixel_values and image_grid_thw, as made by the image processor of a checkpoint (Qwen2VLImageProcessor). The token indices then hold, for each image in order, a run of prod(image_grid_thw[:, i]) ÷ 4 image tokens (image_token_id + 1, <|image_pad|>), between the tokens of <|vision_start|> and <|vision_end|>, as the chat template of the checkpoints writes them. The merged tokens of the image replace the image tokens, and the multimodal rotary embedding gives them the positions of their grid. The processor of a checkpoint, a Qwen3VLProcessor, makes all of these from images and a prompt with a placeholder for each image, and the model takes its output as it is: model(proc(; images, text)).

qwen3_5_moe_for_causal_lm(; vocab_size = 248320, hidden_size = 2048, num_hidden_layers = 40, kwargs...)

Port of transformers' Qwen3_5MoeForCausalLM, the text model of Qwen3.5 MoE (Qwen3.8-2.4T-A95B): the decoder of qwen3_5_moe_text_model, with its keywords, and the output projection lm_head, tied to the token embeddings with tie_word_embeddings. It takes and returns the arrays of qwen3_5_for_causal_lm. The multi-token prediction weights (mtp.*) of the checkpoints are not loaded, as in transformers.

qwen3_5_moe_for_conditional_generation(; text_config, vision_config, image_token_id = 248056,
                                       tie_word_embeddings = false)

Port of transformers' Qwen3_5MoeForConditionalGeneration, the vision-language model of the Qwen3.5 MoE and Qwen3.6 MoE checkpoints (Qwen3.6-35B-A3B), built from its Qwen3_5MoeConfig: the model of qwen3_5_for_conditional_generation, with its inputs, outputs and processor, whose decoder is qwen3_5_moe_text_model of text_config. Videos and the multi-token prediction weights (mtp.*) of the checkpoints are not supported.

qwen3_5_moe_text_model(; vocab_size = 248320, hidden_size = 2048, num_hidden_layers = 40, kwargs...)

Port of transformers' Qwen3_5MoeTextModel, the decoder of Qwen3.5 MoE, built from the fields of its Qwen3_5MoeTextConfig with the same names and defaults, and full_attention_interval, rope_theta, rope_scaling and partial_rotary_factor, which the config reads from config.json. It is the decoder of qwen3_5_text_model, with its inputs and outputs, whose MLPs are sparse mixtures of num_experts gated MLPs of moe_intermediate_size features with a shared expert: each token goes through the num_experts_per_tok experts of largest router probability, weighted by these probabilities renormalised to sum to 1, and through the shared expert, a gated MLP of shared_expert_intermediate_size features weighted by a sigmoid gate. The training options (output_router_logits, router_aux_loss_coef) are ignored.

qwen3_5_text_model(; vocab_size = 248320, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' Qwen3_5TextModel, the decoder of Qwen3.5, built from the fields of its Qwen3_5TextConfig with the same names and defaults, and full_attention_interval, rope_theta, rope_scaling and partial_rotary_factor, which the config reads from config.json. It maps token indices of size (seq_len, batch_size), or a vector, to the hidden states after the final norm, and takes the keywords attention_mask and position_ids (see qwen3_5_decode, which takes input embeddings instead).

Its layers have either a gated delta rule ("linear_attention", Qwen3_5GatedDeltaNet) or causal attention ("full_attention"), as layer_types says, by default every full_attention_interval-th layer. The attention has a sigmoid gate on its output, norms its queries and keys per head and rotates the first head_dim * partial_rotary_factor features of each with the multimodal rotary embedding of rope_parameters["mrope_section"] (only the default rope_type). The RMSNorms scale by 1 + weight. The output of the gated delta rule is gated with SiLU, as in transformers, which ignores output_gate_type: it may be nothing, "silu" or "swish" (Qwen3.8's), and other values are an error.

qwen3_5_vision_model(; depth = 27, hidden_size = 1152, num_heads = 16, patch_size = 16,
                     spatial_merge_size = 2, temporal_patch_size = 2, kwargs...)

Port of transformers' Qwen3_5VisionModel, the vision encoder of Qwen3.5, built from the fields of its Qwen3_5VisionConfig (the vision_config of a Qwen3.5 checkpoint) with the same names and defaults. deepstack_visual_indexes, which config.json may carry, must be empty, as transformers' Qwen3.5 has no deepstack mergers; the rotary embedding is the axial one (rope_type = "axial", or "default").

The model takes (pixel_values, grid_thw), as the image processor of a checkpoint (Qwen2VLImageProcessor) makes them, whose output it also takes as it is (model(ip(images))), and returns the NamedTuple (; last_hidden_state, pooler_output):

• pixel_values, of size (in_channels * temporal_patch_size * patch_size^2, num_patches), is the transpose of transformers' pixel_values: each column is a patch, flattened as torch's (in_channels, temporal_patch_size, patch_size, patch_size) (that is, as a Julia array of size (patch_size, patch_size, temporal_patch_size, in_channels), width first).
• grid_thw, an integer matrix of size (3, num_images) (transformers' grid_thw transposed), has a column (t, h, w) for each image or video: its number of patches along time (frames of temporal_patch_size), height and width. The patches of the images follow one another; within an image, frame after frame, each frame is ordered by windows of spatial_merge_size × spatial_merge_size patches, row-major over the windows and row-major within each window.
• last_hidden_state has size (hidden_size, num_patches). pooler_output, the merged tokens that the language model takes, image after image, has size (out_hidden_size, num_patches ÷ spatial_merge_size^2).

Each frame of each image attends to its own patches only.

qwen3_for_causal_lm(; vocab_size = 151936, hidden_size = 4096, num_hidden_layers = 32, kwargs...)

Port of transformers' Qwen3ForCausalLM, built from the fields of its Qwen3Config with the same names and defaults. transformers derives Qwen3 from Llama's classes, with a Qwen3Attention that normalises each query and key head, so the model is a LlamaForCausalLM (see llama_for_causal_lm for its inputs and outputs). Sliding-window attention (use_sliding_window, layer_types) and rotary embeddings other than the default, "linear" and "llama3" ones error.

qwen3_moe_for_causal_lm(; vocab_size = 151936, hidden_size = 2048, num_hidden_layers = 24, kwargs...)

Port of transformers' Qwen3MoeForCausalLM (Qwen3-30B-A3B, Qwen3-235B-A22B), built from the fields of its Qwen3MoeConfig with the same names and defaults, and head_dim, rope_theta and rope_scaling, which transformers reads from config.json. It takes and returns the arrays of llama_for_causal_lm.

Qwen3 MoE is Qwen3 whose MLP is a sparse mixture of num_experts gated MLPs of moe_intermediate_size features: each token goes through the num_experts_per_tok experts of largest router probability, weighted by these probabilities, renormalised to sum to 1 if norm_topk_prob. The layers in mlp_only_layers (from 0), and all if num_experts = 0, have a dense MLP of intermediate_size features instead, and so do those whose number (from 1) isn't a multiple of decoder_sparse_step. config.json files written by transformers 5 name num_experts num_local_experts. Sliding-window attention (use_sliding_window) and rotary embeddings other than the default, "linear" and "llama3" ones error. The training options (output_router_logits, router_aux_loss_coef) are ignored.

sam3_model(; vision_config = nothing, text_config = nothing,
           geometry_encoder_config = nothing, detr_encoder_config = nothing,
           detr_decoder_config = nothing, mask_decoder_config = nothing)

Port of transformers' Sam3Model, the detector of SAM 3 (Carion et al., 2025), built from its Sam3Config: NamedTuples of the fields of the configs of its parts (other keys are ignored), each nothing for the defaults of transformers. The vision encoder sam3_vision_model of vision_config and the CLIP text encoder clip_text_model_with_projection of text_config feed a DETR encoder and decoder, whose queries the mask decoder turns into masks; box prompts go through a geometry encoder.

The model maps images and text prompts, and optionally box prompts, to the masks, boxes and scores of the objects they describe: see Sam3Model for its inputs and outputs, and Sam3Processor, which create_processor loads, for its inputs and to read its outputs.

sam3_video_model(; detector_config = nothing, tracker_config = nothing)

Port of the detector of transformers' Sam3VideoModel (see Sam3VideoModel), built from its Sam3VideoConfig: the sam3_model of detector_config, a NamedTuple of the fields of its Sam3Config. The tracker of tracker_config is not ported, and the weights of tracker_model and tracker_neck are not loaded.

sam3_vision_model(; backbone_config = nothing, fpn_hidden_size = 256,
                  scale_factors = (4.0, 2.0, 1.0, 0.5))

Port of transformers' Sam3VisionModel, the vision encoder of SAM 3, from the fields of its Sam3VisionConfig: the ViT sam3_vit_model of backbone_config (a NamedTuple of its config's fields; other keys are ignored), and a neck that resamples its patch grid by each of scale_factors into feature maps of fpn_hidden_size channels.

sam3_vit_model(; hidden_size = 1024, intermediate_size = 4736, num_hidden_layers = 32,
    num_attention_heads = 16, num_channels = 3, image_size = 1008, patch_size = 14,
    hidden_act = "gelu", layer_norm_eps = 1e-6, attention_dropout = 0.0,
    window_size = 24, global_attn_indexes = (7, 15, 23, 31),
    layer_scale_init_value = nothing, pretrain_image_size = 336, hidden_dropout = 0.0,
    initializer_range = 0.02, rope_parameters = nothing, rope_theta = nothing)

Port of transformers' Sam3ViTModel, the ViT of SAM 3, from the fields of its Sam3ViTConfig: its layers attend within windows of window_size × window_size patches, but for those of global_attn_indexes (0-based), which attend over the whole grid of an image of image_size, the size the model takes. A 2D axial rotary embedding (rope_theta, from rope_parameters too) gives the positions, scaled to those of a window in the global layers; the learned position embeddings of the pretrain_image_size grid are tiled over larger grids. As in transformers, layer_scale_init_value is not used.

siglip2_model(; text_config = nothing, vision_config = nothing,
              initializer_factor = 1.0)

Port of transformers' Siglip2Model, the image-text dual encoder of the NaFlex checkpoints of SigLIP 2 (see Siglip2Model), built from its Siglip2Config: the siglip_text_model of text_config and the siglip2_vision_model of vision_config, NamedTuples of the fields of their configs (other keys are ignored), each nothing for the defaults of transformers. The projection_size of the text model must be the hidden_size of the vision model.

siglip2_vision_model(; hidden_size = 768, intermediate_size = 3072,
    num_hidden_layers = 12, num_attention_heads = 12, num_channels = 3,
    patch_size = 16, num_patches = 256, hidden_act = "gelu_pytorch_tanh",
    layer_norm_eps = 1e-6, attention_dropout = 0.0, vision_use_head = true)

Port of transformers' Siglip2VisionModel, the vision encoder of the NaFlex checkpoints of SigLIP 2 (see Siglip2VisionModel), built from the fields of its Siglip2VisionConfig: SigLIP's vision transformer, whose patches, flattened, are embedded by a Dense layer, with num_patches learned position embeddings of a square grid, resized to the grid of each image. vision_use_head = false is not supported.

siglip_model(; text_config = nothing, vision_config = nothing, initializer_factor = 1.0)

Port of transformers' SiglipModel, the image-text dual encoder of SigLIP and of the SigLIP 2 checkpoints of fixed resolution (see SiglipModel), built from its SiglipConfig: the siglip_text_model of text_config and the siglip_vision_model of vision_config, NamedTuples of the fields of their configs (other keys are ignored), each nothing for the defaults of transformers. The projection_size of the text model must be the hidden_size of the vision model.

siglip_text_model(; vocab_size = 32000, hidden_size = 768, intermediate_size = 3072,
    num_hidden_layers = 12, num_attention_heads = 12, max_position_embeddings = 64,
    hidden_act = "gelu_pytorch_tanh", layer_norm_eps = 1e-6, attention_dropout = 0.0,
    projection_size = nothing)

Port of transformers' SiglipTextModel, the text encoder of SigLIP and SigLIP 2 (see SiglipTextModel), built from the fields of its SiglipTextConfig (or Siglip2TextConfig, the same): a transformer whose tokens attend to each other in both directions, and whose last token is projected by head to projection_size features (hidden_size by default).

siglip_vision_model(; hidden_size = 768, intermediate_size = 3072,
    num_hidden_layers = 12, num_attention_heads = 12, num_channels = 3,
    image_size = 224, patch_size = 16, hidden_act = "gelu_pytorch_tanh",
    layer_norm_eps = 1e-6, attention_dropout = 0.0, vision_use_head = true)

Port of transformers' SiglipVisionModel, the vision encoder of SigLIP (see SiglipVisionModel), built from the fields of its SiglipVisionConfig: a vision transformer of the patches of patch_size × patch_size pixels of images of image_size × image_size, with learned position embeddings and no class token, whose tokens an attention-pooling head pools. vision_use_head = false, which the configurations of some vision-language models carry, is not supported.

forward_intermediates(model::Union{LlamaForCausalLM, Gemma3ForCausalLM, GraniteForCausalLM, Qwen3_5ForCausalLM},
                      input_ids; indices = nothing, norm = false, stop_early = false,
                      intermediates_only = false, attention_mask = nothing)

The outputs of the decoder layers of a language model (Llama, Qwen2, Qwen3, Qwen3.5, Gemma 3, Granite) for the token indices input_ids of size (seq_len, batch_size), each of size (hidden_size, seq_len, batch_size), or of a vector for a single sequence, each of size (hidden_size, seq_len): the i-th intermediate is transformers' output_hidden_states[i], the output of the i-th layer, except that the last one is not normed unless norm = true. The features are the hidden states after the final norm, as forward_features returns them. indices, norm, stop_early and intermediates_only are those of the vision models, and attention_mask that of the model, as is the position_ids keyword of Qwen3.5. The output of a processor can replace input_ids and attention_mask, as for the model. feature_info gives the hidden_size and the module of each layer.

post_process_instance_segmentation(processor::Sam3Processor, outputs; threshold = 0.3,
                                   mask_threshold = 0.5, target_sizes = nothing)

The objects that SAM 3 segments in each image, from the outputs of a Sam3Model, as transformers' processor of SAM 3 gives them: processor is a Sam3Processor or its image processor. For each image, a NamedTuple with the keys of transformers' result: the scores and boxes of post_process_object_detection, and the masks of the objects, a BitArray of size (width, height, num_objects) where the probability of the mask is above mask_threshold. With the target_sizes of the images, (; height, width) or (height, width), the probabilities are resized bilinearly to them first.

post_process_object_detection(processor::Sam3Processor, outputs; threshold = 0.3,
                              target_sizes = nothing)

The objects that SAM 3 detects in each image, from the outputs of a Sam3Model, as transformers' processor of SAM 3 gives them: processor is a Sam3Processor or its image processor. For each image, a NamedTuple with the keys of transformers' result: the scores of the queries above threshold (the probability of each query times that of the presence of the prompt) and their boxes (4, num_objects), (x1, y1, x2, y2) normalized to the image, or in its pixels with the target_sizes of the images, (; height, width) or (height, width) (such as the original_sizes of the processor's output).

post_process_semantic_segmentation(processor::Sam3Processor, outputs; threshold = 0.5,
                                   target_sizes = nothing)

The semantic segmentation of each image by SAM 3, from the outputs of a Sam3Model, as transformers' processor of SAM 3 gives it: processor is a Sam3Processor or its image processor. For each image, a NamedTuple with the keys of transformers' result with return_segmentation_scores = True: the segmentation, a BitMatrix of size (width, height) where the probability of semantic_seg is above threshold, and the probabilities segmentation_scores, of size (width, height, 1). With the target_sizes of the images, (; height, width) or (height, width), the probabilities are resized bilinearly to them first.

autoencoder_kl_flux2(; latent_channels = 32, block_out_channels = (128, 256, 512, 512), kwargs...)

Port of diffusers' AutoencoderKLFlux2, the autoencoder of FLUX.2, built from the arguments of its constructor (the config.json of the vae of a pipeline), with the same names and defaults. Only DownEncoderBlock2D and UpDecoderBlock2D blocks are supported, which all checkpoints use.

The model maps images of size (width, height, in_channels, batch) to images of the same size, through the mean of the posterior: see encode and decode. Its bn holds the mean and the variance of the latents in patches of patch_size, with which FLUX.2 pipelines normalise them; it takes no part in the model's forward pass.

flux2_transformer_2d_model(; num_layers = 8, num_single_layers = 48, num_attention_heads = 48,
                           attention_head_dim = 128, kwargs...)

Port of diffusers' Flux2Transformer2DModel, the transformer of FLUX.2, built from the arguments of its constructor (the config.json of the transformer of a pipeline), with the same names and defaults. The defaults are those of FLUX.2 [dev]; FLUX.2 [klein] 4B has 5 double- and 20 single-stream blocks of 24 heads.

See Flux2Transformer2DModel for its inputs and output.

autoencoder_kl(; latent_channels = 4, block_out_channels = (64,),
               scaling_factor = 0.18215, shift_factor = nothing, kwargs...)

Port of diffusers' AutoencoderKL, the variational autoencoder of Stable Diffusion, FLUX.1 and Z-Image, built from the arguments of its constructor (the config.json of the vae of a pipeline), with the same names and defaults. Only DownEncoderBlock2D and UpDecoderBlock2D blocks are supported, which all checkpoints use.

The model maps images of size (width, height, in_channels, batch) to images of the same size, through the mean of the posterior: see encode and decode. Pipelines decode z / scaling_factor .+ shift_factor from the latents z they denoise, which the model holds as its fields scaling_factor and shift_factor (nothing for none). sample_size, latents_mean, latents_std and force_upcast are settings of tiling and of other pipelines, which the port ignores.

z_image_transformer_2d_model(; dim = 3840, n_layers = 30, n_heads = 30,
                             cap_feat_dim = 2560, kwargs...)

Port of diffusers' ZImageTransformer2DModel, the transformer of Z-Image, built from the arguments of its constructor (the config.json of the transformer of a pipeline), with the same names and defaults, which are those of Z-Image (Turbo has axes_lens = (1536, 512, 512)). As in diffusers, n_kv_heads is unused. The Omni variant (siglip_feat_dim) is not supported.

See ZImageTransformer2DModel for its inputs and output.

(transformer::Flux2Transformer2DModel)(hidden_states, encoder_hidden_states, timestep,
                                       img_ids, txt_ids, guidance = nothing)

The velocity predicted by the FLUX.2 transformer for the image tokens hidden_states of size (in_channels, image_seq_len, batch), conditioned on the text tokens encoder_hidden_states of size (joint_attention_dim, text_seq_len, batch), at the timesteps (a vector of batch values in [0, 1], the noise levels σ). img_ids and txt_ids are the integer coordinates of the tokens, of size (4, seq_len) (or (4, seq_len, batch), of which the first is used), from which the rotary embeddings are computed. guidance is the guidance scale of guidance-distilled checkpoints (guidance_embeds = true). Returns an array of size (patch_size^2 * out_channels, image_seq_len, batch).

(transformer::ZImageTransformer2DModel)(x, t, cap_feats; patch_size = 2,
                                        f_patch_size = 1)

The output of the Z-Image transformer for the latent images x, of size (width, height, in_channels, batch) or (width, height, frames, in_channels, batch), at the timesteps t (a vector of batch values in [0, 1]: 1 - σ for the noise level σ), conditioned on the captions cap_feats: a vector of batch matrices of size (cap_feat_dim, length), the prompt embeddings of each image without their padding, or an array of size (cap_feat_dim, length, batch) when they have the same length. Returns an array of the size of x, the opposite of the velocity of the flow: Z-Image pipelines step along its negation.

As in diffusers, the patches of each image and the tokens of its caption are padded to a multiple of 32 with the learned x_pad_token and cap_pad_token, and the captions of a batch are then padded to the same length, which the attention masks. patch_size and f_patch_size select one of the pairs of all_patch_size and all_f_patch_size.

encode(vae, x) -> DiagonalGaussianDistribution

The posterior of the latents of the images x, of size (width, height, 3, batch) with values in [-1, 1]: mode gives the latents, of size (width ÷ f, height ÷ f, latent_channels, batch) for the downsampling factor f.

decode(vae, z)

The images, with values about [-1, 1], of the latents z.

DiagonalGaussianDistribution(parameters)

diffusers' posterior of an autoencoder: a Gaussian with the mean and the log-variance (clamped to [-30, 20]) that are the two halves of the channels of the encoder's output parameters. mode returns its mean, and sample draws from it.

mode(posterior::DiagonalGaussianDistribution)

The mean of the posterior of an autoencoder.

sample([rng], posterior::DiagonalGaussianDistribution)

A draw from the posterior of an autoencoder, with noise from rng (generated on the CPU).

FlowMatchEulerDiscreteScheduler

diffusers' Euler sampler of flow-matching models, as configured by the scheduler_config.json of a pipeline. Unlike diffusers' scheduler it keeps no state: set_timesteps returns the schedule, and step takes the noise levels of a step.

set_timesteps(scheduler, num_inference_steps; sigmas = nothing, mu = nothing) -> (timesteps, sigmas)

The Float32 timesteps of the denoising steps (sigmas .* num_train_timesteps) and their noise levels, with a final 0. The sigmas are shifted by mu with use_dynamic_shifting, which then requires it, or else by the scheduler's shift. By default they are evenly spaced from 1 to the smallest noise level of training.

step(scheduler, model_output, sigma, sigma_next, sample)

The sample of an Euler step of the flow model_output from the noise level sigma to sigma_next, computed in at least Float32 and returned in the element type of model_output.

compute_empirical_mu(image_seq_len, num_steps)

The shift of the noise levels of FLUX.2 for image_seq_len image tokens and num_steps steps, fitted by Black Forest Labs.

get_vision_features(model::Sam3Model, pixel_values)

The output of the vision encoder of SAM 3 on the images pixel_values (see Sam3VisionModel). Pass it to the model in place of the images to reuse it for several prompts.

qwen3_5_decode(model::Qwen3_5TextModel, inputs_embeds, position_ids; attention_mask = nothing)

The hidden states (hiddensize, seqlen, batchsize) after the final norm of the Qwen3.5 text model for the input embeddings `inputsembeds(hidden_size, seq_len, batch_size), transformers'Qwen35TextModel.forward(inputsembeds=...).positionids` are the temporal, height and width positions of the tokens for the rotary embedding, an integer array (3, seqlen, batchsize) or (3, seqlen, 1) of 0-based positions (torch's position_ids (3, batchsize, seqlen) permuted), or nothing for 0:seq_len-1 on each row, those of text. attention_mask (seqlen, batchsize) marks the tokens of a batch padded on the right, as for the token indices.

EfficientNetBuilder(; output_stride = 32, pad_type = "", round_chs_fn = round_channels,
                    se_from_exp = false, act_layer = nothing, norm_layer = BATCHNORM2D,
                    aa_layer = nothing, se_layer = nothing, drop_path_rate = 0.0,
                    layer_scale_init_value = nothing)

Builds the stages of blocks from decoded block arguments. Calling it as builder(in_chs, block_args) returns a vector of Chains, and leaves the channel count of the last block in builder.in_chs. se_layer(channels; act_layer, rd_ratio) creates the squeeze-and-excitation module of a block. aa_layer (anti-aliasing, see create_aa) is supported by the cn, er, uir and mqa blocks, and layer_scale_init_value applies to the uir and mqa blocks of MobileNetV4.

MixedConv2d(in_chs, out_chs, kernel_sizes; stride, dilation, padding, depthwise, bias)

timm.layers.MixedConv2d (MixConv): the channels are split into one group per kernel size, each convolved with its own kernel size, and concatenated. A ModuleDict in timm, so the convs' keys are 0.weight, 1.weight, ... with no segment for the container.

CondConv2d(in_chs, out_chs, kernel_size; num_experts, stride, dilation, padding, groups, bias)

timm.layers.CondConv2d: a conv whose kernel (and bias) is a per-sample mixture of num_experts kernels, called as m(x, routing_weights) with routing weights of size (num_experts, batch). weight is torch's (num_experts, params) reversed: column e is the flattened kernel of expert e. The samples are convolved at once, as one grouped conv.

MultiheadAttention

PyTorch's nn.MultiheadAttention with batch_first = True, keys and values of the size of the queries and no extra biases, as the attention-pooling heads of SigLIP use it: the queries, keys and values are projected by the blocks of in_proj_weight, of size (embed_dim, 3 * embed_dim) (torch's (3 * embed_dim, embed_dim)), and in_proj_bias. Called on (query, key, value) of size (embed_dim, len, batch), with the keys of each sequence limited to its first kv_lengths, it returns (attn_output, nothing), as torch returns the output and the attention weights, which aren't computed.

timestep_embedding(timesteps, embedding_dim; flip_sin_to_cos = false,
                   downscale_freq_shift = 1, scale = 1, max_period = 10_000)

diffusers' get_timestep_embedding: the sinusoidal embeddings of a vector of (possibly fractional) timesteps, of size (embedding_dim, length(timesteps)), on the device of timesteps. As in diffusers, they are computed in at least Float32: in BFloat16, the angles of timesteps near 1000 would be off by several radians.

AdaLayerNormContinuous(embedding_dim, conditioning_embedding_dim; elementwise_affine = true,
                       eps = 1e-5, bias = true)

diffusers' adaptive LayerNorm of tokens (embedding_dim, seq_len, batch), scaled and shifted by a projection of the conditioning embedding (conditioning_embedding_dim, batch).

create_conv2d(in_chs, out_chs, kernel_size; stride, dilation, padding = "", groups, depthwise, bias = false, num_experts = 0)

timm.layers.createconv2d. Sizes are Ints or torch-order (h, w) pairs; a vector of kernel sizes gives a MixedConv2d and `numexperts > 0a [CondConv2d](@ref).paddingis""(PyTorch-style symmetric),"same"(TF-style, aConv2dSameunless it is static),"valid"`, or an explicit torch padding.

decode_arch_def(arch_def, depth_multiplier = 1.0; depth_trunc = "ceil",
                experts_multiplier = 1, fix_first_last = false, group_size = nothing)

Decode the block definition strings of the stages of arch_def (a vector of vectors of strings) into a vector of vectors of block arguments. depth_multiplier is a number or one per stage; experts_multiplier scales the number of CondConv experts.

chunk_gated_delta_rule(q, k, v, g, beta; chunk_size = 64)

transformers' torch_chunk_gated_delta_rule with use_qk_l2norm_in_kernel = true, for the queries and keys q and k (headkdim, heads, seqlen, batch), the values v (headvdim, heads, seqlen, batch), the log decays g and the rates beta (heads, seqlen, batch). For each head, with q and k normalised and q scaled by `headkdim^-0.5, the stateS` (headvdim, headk_dim) starts at zero and at each token becomes S = exp(g) * S then S += beta * (v - S * k) * kᵀ, and the output is S * q.

As in transformers, the sequence is padded to chunks of chunk_size tokens, computed in parallel but for the state carried from one chunk to the next, in at least Float32. Each chunk solves a unit triangular system, here by inverting its matrix by blocks, which runs on a GPU and is differentiable. The output has the element type of v.

moe_experts(gate_up, down, act, x, top_k_index, top_k_weights)

The experts of a sparse mixture of experts, as transformers' MixtralExperts: each token, a column of x (hidden, n), goes through its experts top_k_index (k, n), whose outputs down * (act(gate) .* up) are summed with the weights top_k_weights (k, n). The stacked weights of the experts are torch's gate_up_proj (numexperts, 2 * intermediate, hidden) and `downproj(num_experts, hidden, intermediate), sogateup` has size (hidden, 2 * intermediate, numexperts) and down (intermediate, hidden, num_experts).

The k * n pairs of a token and an expert are sorted by expert, so that each projection is one NNlib.grouped_mul over the groups of different sizes, without padding. Each token then gathers back the outputs of its experts. Unlike a scatter, this needs no atomic additions, which are nondeterministic, and slow in BFloat16 on GPUs before sm_90.

pil_resize(img, (width, height), filter = :bicubic; torchvision = false)
    -> Array{UInt8, 3}

img, a (width, height, channels) array of UInt8, resized as Pillow's Image.resize((width, height), filter) resizes an 8-bit image, bit for bit. filter is one of :nearest, :box, :bilinear, :hamming, :bicubic and :lanczos.

With torchvision = true, for :bilinear and :bicubic, it is resized bit for bit as torchvision resizes a uint8 tensor on an x86 CPU, with antialias = True. torch's variant of the algorithm, from Pillow-SIMD, always resizes the width first, and rounds the weights of each pass to as many bits as int16 holds. (For :bilinear without AVX2, and on a GPU, torchvision resizes in floating point instead.)

checkpoint_prefix(model, ckpt::Checkpoint) -> String

The prefix of the keys under which the checkpoint ckpt holds the weights of model: "" (the default), or the name of a module that some checkpoints of model nest its weights in, and that the original library removes from their keys as it loads them (transformers' PrefixChange: "text_model" for the CLIPTextModel checkpoints saved before transformers 5).

checkpoint_tensor(model, param::TorchParam, x) -> AbstractArray

The tensor x that a checkpoint holds for param, a parameter of model, in the layout of param.array, as load_state! is about to check and copy it. By default it is x itself. A method does what the original library's checkpoint filter does to that tensor (timm's checkpoint_filter_fn resamples the position embedding of a Vision Transformer saved for another image size), and returns x itself where nothing changes. A result of another shape than param.array is a mismatch for the strict loader.

load_checkpoint!(model, ckpt::Checkpoint; skip, ignore, initializers)

Load the checkpoint ckpt into model, as instantiate does after building it, with the keywords of load_state!: the weights under checkpoint_prefix(model, ckpt) by default, each tensor through checkpoint_tensor. A method for a model type loads the checkpoints of another layout, as transformers loads those of a base model into a model with a head. initializers maps the arrays of model to the functions of their size that initialize them, for the parameters that such a method leaves out of the checkpoint.

AbstractProcessor

A processor, as create_processor returns it: an image processor, or a processor that combines processors and a tokenizer. Each is a plain immutable struct whose fields are the attributes of the Python class (its settings, and the processors it combines), with a keyword constructor whose defaults are those of the Python class, and a provenance field that records where each setting comes from, for show.

AbstractImageProcessor <: AbstractProcessor

A processor of images, called as ip(images) on one image or a vector of them (see image_array).

@processor struct T <: AbstractProcessor ... end
@processor (components...,) struct T <: AbstractProcessor ... end

Base.@kwdef for a processor class, whose constructors make the processor from standardize_settings of its settings, so that a processor made by hand fails as early as a loaded one. components names the fields that hold the processors it combines, each of type Union{Nothing, C} with the default nothing, which create_processor loads as a processor of class C from the same repository; a component that isn't a C is an error. The component tokenizer, of type Any, is the tokenizer of the repository: HuggingFaceTokenizers.jl's Tokenizer of its tokenizer.json, or nothing without HuggingFaceTokenizers or without the file. With components, T(components...; kwargs...) also makes it from them, e.g. Sam3Processor(image_processor, tokenizer; point_pad_value = -10).

register_processor!(name, T; aliases = (), model_types = ())

Register the processor class T as the port of transformers' (or another library's) class name, which configurations name in processor_class or image_processor_type, and of the classes aliases. The suffixes Fast and Pil of class names, those of transformers' former image processors and of its PIL backend, are dropped before lookup. model_types are the model_types of config.json whose processor is T (AutoProcessor's PROCESSOR_MAPPING), for repositories whose processor files don't name a class.

ignored_keys(T) -> NamedTuple

The keys of a configuration that the processor class T knows but doesn't read, as key = "reason", e.g. (; mask_size = "only resizes the segmentation maps of training"). Other keys that aren't settings of T warn once. An override of an ignored key is an error that gives the reason.

legacy_keys(T) -> NamedTuple

Former names of settings of the processor class T, as key = path, where path is the name of a setting or the path to an entry of a nested one, e.g. (; min_pixels = (:size, :shortest_edge)). The value of a legacy key replaces the one at its path, in files and overrides, and drops the entry of the legacy key from the nested setting, as transformers does.

standardize_settings(T, settings::NamedTuple) -> NamedTuple

The settings of a processor of class T as its constructor stores them, from settings, the values it is given for the fields that are neither components nor provenance: the forms of a setting that the Python library also takes, converted (an integer size to (; height, width)), and an ArgumentError for any other value, or for one that the port doesn't implement (do_center_crop = true), naming the supported values. An error whose message names a setting in backticks, as Metalhead's errors do, also names the file that set it when the processor is loaded. The default returns settings as they are, which the constructor converts to the types of the fields.

instantiate_processor(format::AbstractConfigFormat, files, config; kwargs...)
    -> Union{Nothing, AbstractProcessor}

The processor of a repository whose config.json, parsed as config, is in format and carries the preprocessing of its checkpoint (timm's pretrained_cfg), with the overrides kwargs; or nothing (the default) for create_processor to find the processor files of the repository as transformers' AutoProcessor does. files reads the other files of the repository.

check_inputs(model, inputs::NamedTuple, required; hint = "")

Throw if the NamedTuple inputs of model(inputs) lacks keys of required, naming them and the keys it has, so that the inputs of another model fail at the call rather than deep inside. Every model(inputs::NamedTuple) method calls it on the keys it can't do without.

torch_params(model; prefix = "") -> (params::Vector{TorchParam}, aliases::Vector{Pair{String,String}})

Enumerate the parameters and buffers of model with their PyTorch state_dict keys. An array reached twice (tied weights) is listed once; later keys are returned as aliases.

TorchParam(key, array, kind)

A parameter or buffer of a Flux model, with its PyTorch state_dict key. kind selects the transform from the checkpoint's Julia-layout tensor: :plain (none), :dense (transpose) or :conv (flip the spatial dimensions).