Getting started

For a worked example with photographs, a custom image dataset and transfer learning, see Working with vision models.

Creating a model

create_model builds a ported model. Its first argument names the model in one of four ways:

using Metalhead

model = create_model("timm/resnet50.a1_in1k")                     # weights from the Hub
model = create_model("resnet50")                                  # random weights, no download
model = create_model("timm/resnet50.a1_in1k"; pretrained = false) # that repository's architecture, random weights

A repository or a directory names a checkpoint, whose weights create_model loads; a bare architecture has random weights unless pretrained = true. Metalhead downloads and caches the repository's config.json and safetensors weights (sharded checkpoints too). The cache is shared with Python's huggingface_hub (~/.cache/huggingface/hub, or HF_HUB_CACHE), and so are HF_TOKEN for private or gated repositories and HF_HUB_OFFLINE to use only cached files. The model list shows the checkpoints each architecture was validated against. Bare architecture names use the first of them. Validated checkpoints load at the commit they were validated at, other repositories from their main branch; revision picks a branch, tag or commit:

model = create_model("timm/resnet50.a1_in1k"; revision = "main")

The other keyword arguments are the original library's model arguments, such as timm's num_classes, drop_rate or drop_path_rate. They override the repository's configuration. Use eltype to build the model in another precision (Float16, Flux.BFloat16, Float32 or Float64).

list_models lists the ported architectures, or with pretrained = true the validated checkpoints:

list_models()
list_models(; pretrained = true)

Use byfamily = true to return a dictionary keyed by family identifiers, such as "resnet" and "xception_aligned":

families = list_models(; byfamily = true)
keys(families)
families["resnet"]
list_models(; byfamily = true, pretrained = true)["resnet"]

Browse the model list for architectures grouped by family and links to their validated checkpoints.

Inputs

Image models take Flux's layout: width × height × channels × batch, in Float32 unless you chose another eltype. PyTorch's (N, C, H, W) is the same memory layout with the dimensions reversed. Sizes in timm configurations, such as input_size = (3, 224, 224), are in PyTorch order. The processor of a checkpoint makes these inputs from images (see below).

The model is returned in test mode: dropout is off and batch normalisation uses its running statistics. Call Flux.trainmode!(model) before training.

Classifying an image

A checkpoint expects its images prepared as in its evaluation: resized, cropped and normalised with the settings of its configuration. create_processor reads them from the same repository and revision as the weights, and returns the processor of the checkpoint, which applies them. For a timm checkpoint, it is timm's eval transform, a Timm.EvalTransform:

using Metalhead, Flux, FileIO, ImageCore

repo = "timm/convnext_tiny.fb_in1k"
model = create_model(repo)
transform = create_processor(repo)

img = load(download("https://upload.wikimedia.org/wikipedia/commons/2/26/YellowLabradorLooking_new.jpg"))
x = transform(img)                # 224 × 224 × 3 × 1
probs = softmax(model(x))
labels = readlines(download("https://raw.githubusercontent.com/pytorch/hub/master/imagenet_classes.txt"))
labels[sortperm(vec(probs); rev = true)[1:5]]
# "Labrador retriever", "kelpie", "dingo", "golden retriever", "Chesapeake Bay retriever"

FileIO reads the photo with ImageIO, which must be installed, as a height × width matrix of colours. With ImageCore loaded (Images.jl loads it too), processors take such matrices, and turn them into the width × height × channels arrays of Flux. Without it, pass an array of that layout, of UInt8 from 0 to 255 or of reals from 0 to 1. A vector of images makes a batch.

The transform resizes the shorter side of the image to floor(224 / 0.875) = 256 pixels with Pillow's bicubic filter, crops the centre 224 × 224 pixels, and normalises each channel with mean and std. These are the values that timm computes from the same pixels as a PIL image, bit for bit (the pixels of a JPEG decoded by ImageIO can differ slightly from Pillow's). show prints each setting and where it comes from:

transform
# EvalTransform
#   input_size = (3, 224, 224)    config.json
#   interpolation = "bicubic"     config.json
#   mean = (0.485, 0.456, 0.406)  config.json
#   std = (0.229, 0.224, 0.225)   config.json
#   crop_pct = 0.875              config.json
#   crop_mode = "center"          config.json

Many timm checkpoints are evaluated at a larger size than they were trained at, given by test_input_size and test_crop_pct in their configuration. use_test_size = true takes these, as timm's resolve_data_config(use_test_size = True) does. Other keyword arguments override a setting by name, also in natural forms (input_size = 256 for (3, 256, 256), mean = 0.5 for every channel, interpolation = :bilinear):

transform = create_processor(repo; use_test_size = true)
size(transform(img))              # (288, 288, 3, 1)
transform = create_processor(repo; crop_pct = 1.0)
transform.crop_pct                # 1.0, shown as "override (config.json: 0.875)"

A setting that the port doesn't implement, such as timm's crop_mode = "border", is an error, and so is a keyword that isn't a setting. Metalhead.Timm.pretrained_cfg returns the settings as the configuration stores them.

Processors

timm's transform is one kind of processor. Each checkpoint has one, which applies the settings stored in its repository as its Python library does: create_model makes the forward pass match PyTorch, and create_processor the whole path from raw input to result. It reads the files of a branch at one commit, that of the first file it finds, and offline or without network those of the cache.

timm's transform returns an array of size width × height × channels × batch. The other processors return a NamedTuple with the keys of transformers (input_ids, pixel_values, ...), in Julia's layout, with token indices that start at 1, which the model takes as it is: model(inputs). The processors that take text also decode token indices, with HuggingFaceTokenizers' decode(proc, indices) (see Language models). SAM 3's processor also turns the outputs of the model back into results (see Segmenting objects). A diffusers pipeline has no processor: it prepares its inputs itself, with its tokenizer pipe.tokenizer.

Metalhead ports no tokenizer, the only part of the path from input to result that isn't Julia. The tokenizer of a repository is a Tokenizer of HuggingFaceTokenizers.jl, which calls Hugging Face's tokenizers library through Python on the repository's tokenizer.json, with the padding token of its tokenizer_config.json. create_processor loads it after using HuggingFaceTokenizers, as the tokenizer of a processor that takes text: the TextProcessor of a language model, or those of CLIP, Qwen3.5 and SAM 3, which without it prepare images only. Its token ids start at 0, the token indices of the models at 1.

The settings are read strictly, for every processor: one that the port doesn't implement is an error, and an unknown key in a file a warning.

Features and the classifier

Image models follow timm's split into a feature extractor and a head:

x = rand(Float32, 224, 224, 3, 2)
features = forward_features(model, x)                       # e.g. 7 × 7 × 768 × 2 for ConvNeXt-T
pooled = forward_head(model, features; pre_logits = true)   # 768 × 2
logits = forward_head(model, features)                      # 1000 × 2

Language models have the same interface, with their hidden states as features (see Language models).

Multi-scale features

Segmentation and depth heads take the feature maps of several stages. forward_intermediates returns them, like timm's, and feature_info gives their channels and strides:

resnet = create_model("resnet50")
feature_info(resnet)              # (num_chs = 64, reduction = 2, module_name = "act1"), ...
maps = forward_intermediates(resnet, x; indices = 2:5, intermediates_only = true)
size.(maps)                       # 56 × 56 × 256 × 2, 28 × 28 × 512 × 2, ...

indices are 1-based positions in feature_info, or an integer k for the last k. stop_early = true stops at the last selected tap; norm = true applies a final norm where the architecture supports it. Convolutional models accept output_fmt = "NCHW" (WHCN arrays in Julia).

Feature extraction covers ResNet and its variants (including Res2Net), ConvNeXt, ViT, the MLP mixers, DenseNet, VGG, Inception v3/v4, Inception-ResNet-v2, aligned Xception, and all ported EfficientNet and MobileNet families (including MixNet, MNASNet, FBNet and LCNet). ConvMixer has no upstream feature tap table and is unsupported. The create_model(...; features_only = true, out_indices = ...) convenience interface is not implemented; use forward_intermediates on the existing model.

Taps follow timm's definitions: VGG includes a reduction-1 tap, ConvNeXt starts at reduction 4, and Xception taps inside its strided blocks before downsampling. Dilated EfficientNet and Xception models can have several taps with the same reduction.

Vision transformers return the tokens of their blocks as feature maps of the patch grid (output_fmt = "NLC" keeps them as tokens), and with dynamic_img_size = true accept any input size that is a multiple of the patch size:

dinov2 = create_model("vit_small_patch14_dinov2"; pretrained = true, dynamic_img_size = true)
maps = forward_intermediates(dinov2, rand(Float32, 224, 308, 3, 1); indices = 4,
    intermediates_only = true)    # 16 × 22 × 384 × 1 each

Fine-tuning

Pass a new num_classes to load the pretrained weights with a freshly initialised classifier:

using Flux

model = create_model("timm/convnext_tiny.fb_in1k"; num_classes = 10)
Flux.trainmode!(model)

loss(m, x, y) = Flux.logitcrossentropy(m(x), y)
data = [(rand(Float32, 224, 224, 3, 4), rand(1:10, 4))]  # labels in 1:num_classes
opt_state = Flux.setup(AdamW(1e-4), model)
Flux.train!(loss, model, data, opt_state)

With num_classes = 0 the model has no classifier and returns the pooled features.

Model structure

A ported model mirrors the PyTorch module tree: its fields have the names of the PyTorch submodules, and nn.Sequentials are Chains with Julia index i for PyTorch index i - 1. PyTorch's model.layer1[0].conv1 is model.layer1[1].conv1:

model = create_model("resnet50")
model.layer1[1].conv1     # Conv((1, 1), 64 => 64, bias=false)
model.fc                  # Dense(2048 => 1000)

The layers are ordinary Flux layers, so Flux.freeze!, Flux.state and Flux.loadmodel! work as usual.

GPU

Load CUDA.jl and cuDNN, or another GPU backend, and move the model and its inputs to the device. See the Flux GPU guide.

using Flux, Metalhead, CUDA, cuDNN

device = gpu_device()
model = create_model("timm/convnext_tiny.fb_in1k") |> device
logits = model(device(rand(Float32, 224, 224, 3, 8)))