borealis-embed

Model Summary

NbAiLab/borealis-embed-212m is a 212M parameter sentence-transformers embedding model (98M embedding parameters) based on an adapted Gemma3Text architecture. The model is pretrained from scratch on the open subset of our Aurora 2604 data and fine-tuned as a sentence transformer with mean pooling on subsets from NbAiLab/mnli-norwegian, DDSC/nordic-embedding-training-data, sentence-transformers/parallel-sentences-opus-100, lightonai/embeddings-pre-training, and internal newspaper articles. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval. With a longer max sequence length than its predecessors (nb-sbert and nb-sbert-v2), the model can handle chunks of text up to 8192 tokens long.

Model Details

Model Description

• Model Type: Sentence Transformer
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Datasets:
  • NbAiLab/mnli-norwegian
  • A subset of retrieval triplets from DDSC/nordic-embedding-training-data
  • Norwegian to English pairs from sentence-transformers/parallel-sentences-opus-100
  • Norwegian Wikipedia articles from lightonai/embeddings-pre-training
  • An internal dataset created from Norwegian newspapers. The Norwegian government has entered into an agreement, through the National Library of Norway, to access copyrighted press material via Kopinor on behalf of the Norwegian Media Businesses' Association (MBL), enabling the lawful training, development, maintenance, and public release of Norwegian language models. The agreement covers copyright-protected press publications published in Norway up to one year from the date of publication of the model, effectively creating a rolling cutoff date rather than a fixed historical cutoff. For this release, the cutoff date is January 1, 2025.
• Language: Norwegian Bokmål, Norwegian Nynorsk, Swedish, Danish, English
• License: NB-license

EU AI Act

This release is a non-generative encoder model whose outputs are vectors/scores rather than language or media. Its intended functionality is limited to representation, retrieval, ranking, or classification support. On that basis, the release is preliminarily assessed as not falling within the provider obligations for GPAI models under the EU AI Act definitions, subject to legal confirmation if capability scope or marketed generality changes. For more information, see the Model Documentation Form.

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'AustralisModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("NbAiLab/borealis-embed-212M")
# Run inference
queries = [
    '- Who is he?',
]
documents = [
    '- Hvem er han?',
    'Banning.',
    'Nei, det har hun ikke.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.7765, 0.2560, 0.2525]])

Evaluation

We evaluated this model on two benchmarks: the Scandinavian Embedding Benchmark (SEB) and the Long-context Retrieval Benchmark (LongEmbed) from MTEB.

During pretraining, max sequence length of the model was extended to 8192 tokens. However, during the fine-tuning process, the model is trained with a sequence length 2048. The evaluation results on LongEmbed show that the model has retained the length extension from pretraining and confirms that the model can support up to 8192 tokens.

Metrics

MTEB (Scandinavian)

"Scandinavian text embedding quality covering Danish, Swedish, Norwegian Bokmål, and Nynorsk and spanning classification, clustering, retrieval as well as bitext tasks across dialects or written forms."