BERT, XLM-RoBERTa and ModernBERT

The text encoders of transformers: BertModel (MiniLM, the BGE and GTE embedding models, ...), XLMRobertaModel (the multilingual E5 models, BGE-M3 and the BGE rerankers' backbone, ...) and ModernBertModel (ModernBERT and the models built on it). They map the token indices of their processor, the TextProcessor of the repository, to the hidden states of the tokens; a head on top gives the logits of a classifier, such as a cross-encoder that ranks passages, or of a masked language model.

using Metalhead, HuggingFaceTokenizers

model = create_model("answerdotai/ModernBERT-base")   # a ModernBertForMaskedLM
proc = create_processor("answerdotai/ModernBERT-base")
inputs = proc(["The capital of France is [MASK].", "Dogs bark."])
hidden = forward_features(model, inputs)    # 768 × seq_len × 2: its ModernBertModel's output
logits = model(inputs)                      # 50368 × seq_len × 2

reranker = create_model("BAAI/bge-reranker-base")
proc = create_processor("BAAI/bge-reranker-base")
inputs = proc(["what is a panda?", "what is a panda?"];
    text_pair = ["The giant panda is a bear native to China.", "Paris is the capital of France."])
reranker(inputs)                            # 1 × 2 scores, the first higher

The models take input_ids and the attention_mask that excludes the padding of a batch (and BERT's token_type_ids), or the output of the processor as it is. The processors of these repositories return the keys of transformers' tokenizers: token_type_ids too for BERT's, and with text_pair the two texts of a cross-encoder joined by the tokenizer's template. Pooling the hidden states into one embedding per text, as sentence-transformers does, is not part of these models. The repositories of embedding models (all-MiniLM-L6-v2, bge-small-en-v1.5, multilingual-e5-base, gte-modernbert-base, ...) have a modules.json: create_model builds the sentence-transformers model over them (see Sentence transformers), and create_processor has no processor for it. Their BertModel, XLMRobertaModel or ModernBertModel is the one that the config.json names, which the references of these repositories validate.

Models

Metalhead.Transformers.TextEncoder — Type

TextEncoder

The base models of the text encoders BERT, XLM-RoBERTa and ModernBERT, which map token indices input_ids (seq_len, batch), 1-based, to the hidden states of each token: BertModel, XLMRobertaModel and ModernBertModel.

(model::TextEncoder)(input_ids; attention_mask = nothing, token_type_ids = nothing)
(model::TextEncoder)(inputs::NamedTuple)

The last_hidden_state of transformers' model, of size (hidden_size, seq_len, batch), the hidden states of each token after the last layer (and the final norm of ModernBERT), as the decoder models return theirs. Every token attends to every other, in both directions. A batch of sequences of different lengths, padded on the right, takes the tokenizer's attention_mask (seqlen, batch), which keeps the padding out of the keys; the outputs at the positions of the padding are not meaningful. `tokentypeids` (seqlen, batch) are the segment ids of transformers, 0 or 1, and default to 0 (ModernBERT takes none). A vector input_ids is a single sequence, whose output has no batch dimension. The output of the processor of a repository, a TextProcessor, can replace all of them: model(proc(texts)).

forward_features is the same call, and forward_intermediates returns the output of each layer (transformers' hidden_states without the embeddings; ModernBERT's last one is normed only with norm = true), with the keyword arguments of the vision models' (indices, stop_early, intermediates_only) and those of the model. feature_info gives the module of each layer. The BERT and XLM-RoBERTa models also have transformers' pooler_output (hiddensize, batch), the first token through the dense layer and tanh of the pooler: `forwardhead(model, hiddenstates), which errors for a model built without a pooler (addpooling_layer = false`).

BERT, XLM-RoBERTa and ModernBERT

Models

Builders