Sentence transformers

SentenceTransformer

A sentence-transformers model, as create_model builds it from a repository with a modules.json, such as "Qwen/Qwen3-Embedding-0.6B": the modules of its pipeline, applied in order to a NamedTuple of features, as sentence-transformers' SentenceTransformer, with the settings of its config_sentence_transformers.json.

• The Transformer module, model[1], holds the ported transformers model (model[1].auto_model) and its tokenizer, and adds the token_embeddings of the texts.
• Pooling makes the sentence_embedding of each text from them, which Dense modules project and Normalize normalizes.
• prompts, the prompts of the model by name, such as the instruction of the queries of Qwen3-Embedding, and default_prompt_name, the name of the prompt of encode (nothing for none);
• similarity_fn_name, the similarity of embeddings of similarity: "cosine" (the default), "dot", "euclidean" or "manhattan";
• truncate_dim, the number of features that encode keeps (nothing for all), for models trained to work with fewer (Matryoshka embeddings).

The keyword arguments of create_model override these settings, e.g. create_model(repo; truncate_dim = 256), and prompts adds to the model's.

model(features) applies the modules to the NamedTuple of features that Metalhead.SentenceTransformers.preprocess makes of texts, with token indices input_ids and their attention_mask (from 1, of size (seq_len, batch)), and returns them with the outputs of each module: token_embeddings and sentence_embedding, of size (dim, batch). encode, encode_query and encode_document take the texts.

using Metalhead, HuggingFaceTokenizers
model = create_model("Qwen/Qwen3-Embedding-0.6B")
queries = encode_query(model, ["What is the capital of China?", "Explain gravity"])
documents = encode_document(model, ["The capital of China is Beijing.",
    "Gravity is a force that attracts two bodies towards each other."])
similarity(model, queries, documents)   # 2 × 2, the largest on the diagonal

encode(model::SentenceTransformer, texts; prompt_name = nothing, prompt = nothing,
       batch_size = 32, normalize_embeddings = false, truncate_dim = model.truncate_dim)

The embeddings of texts, a string or a vector of strings, as sentence-transformers' encode: a matrix of size (dim, length(texts)), or a vector for one string, on the CPU. encode is HuggingFaceTokenizers.jl's, which using HuggingFaceTokenizers brings in, and which tokenizes them with the tokenizer of the model.

The texts are encoded with the string prompt before each, or with the prompt named prompt_name of model.prompts, or with that of model.default_prompt_name if neither is given; in batches of batch_size texts of similar lengths, on the device of the model. The embeddings keep their first truncate_dim features, and are then normalized with normalize_embeddings = true.

encode_query(model::SentenceTransformer, texts; kwargs...)

encode for queries, as sentence-transformers' encode_query: with the prompt named "query" if the model has one, unless prompt or prompt_name is given.

encode_document(model::SentenceTransformer, texts; kwargs...)

encode for documents, as sentence-transformers' encode_document: with the first prompt named "document", "passage" or "corpus" that the model has, unless prompt or prompt_name is given.

similarity(model::SentenceTransformer, a, b)

The similarities of the embeddings a and b (matrices of size (dim, n), or vectors) by the similarity function of the model, model.similarity_fn_name, as sentence-transformers' similarity: a matrix of size (n_a, n_b), the cosine similarities, dot products, or negative Euclidean or Manhattan distances of their columns.

preprocess(transformer::Transformer, texts; prompt = nothing)

The inputs of a SentenceTransformer for a vector of strings texts, as sentence-transformers' preprocess makes them, with the string prompt prepended to each: their token indices, 1-based, truncated to max_seq_length and padded on the right with the padding token of the tokenizer, of size (seq_len, batch), as input_ids; their attention_mask; and with a non-empty prompt its length in tokens, prompt_length, which the pooling leaves out where it doesn't include the prompt.

preprocess(model::SentenceTransformer, texts; prompt = nothing)

The features of the vector of strings texts that model(features) takes, made by its Transformer module, with the string prompt before each, as sentence-transformers' preprocess. Needs using HuggingFaceTokenizers.

Transformer

sentence-transformers' Transformer module, the first of a SentenceTransformer: its auto_model, the ported transformers model that transformers' AutoModel loads from the repository's config.json, without a head (the decoder of Qwen3-Embedding's Qwen3ForCausalLM), and its tokenizer, HuggingFaceTokenizers.jl's Tokenizer of the repository's tokenizer.json (a reference to the file before using HuggingFaceTokenizers).

m(features) adds the token_embeddings of the input_ids of the NamedTuple features, the last hidden states of auto_model, of size (hidden_size, seq_len, batch), with their attention_mask.

Texts are encoded with the special tokens of the tokenizer, truncated to max_seq_length tokens (on the truncation_side of tokenizerconfig.json), and padded on the right. The `maxseqlength` of sentencebertconfig.json takes precedence over the `modelmaxlength` of tokenizerconfig.json (where sentence-transformers 5 saves it), which is capped at the max_position_embeddings of config.json, as sentence-transformers does. do_lower_case is supported where the normalizer of the tokenizer already lowercases.

Pooling(embedding_dimension; pooling_mode = "mean", include_prompt = true)

sentence-transformers' Pooling module: the sentence_embedding of each text, of size (embedding_dimension * length(pooling_mode), batch), from the token_embeddings of size (embedding_dimension, seq_len, batch) and the attention_mask of a NamedTuple of features, by each mode of pooling_mode (a string or a tuple of them, whose embeddings are concatenated): "cls", the first token; "lasttoken", the last one; "mean", "max", and "mean_sqrt_len_tokens", the sum divided by the square root of the number of tokens; "weightedmean", the mean weighted by the position of each token (from 1). The padding, on the right or on the left, is left out, and so are the first prompt_length tokens of the features with include_prompt = false.

Dense(in => out; bias = true, activation_function = tanh,
      module_input_name = "sentence_embedding", module_output_name = module_input_name,
      use_residual = false)

sentence-transformers' Dense module: the feature module_input_name of a NamedTuple of features, through the Linear layer linear and activation_function, as the feature module_output_name. With use_residual, the input is added to the output, through the Linear layer residual (without bias) where in != out. Its weights are those of the model.safetensors of its subfolder.

Normalize(; module_input_name = "sentence_embedding",
          module_output_name = module_input_name)

sentence-transformers' Normalize module: the feature module_input_name of a NamedTuple of features divided by its Euclidean norm along the first dimension (the features of each text or token), as torch's normalize(x, p = 2, dim = -1), as the feature module_output_name.