nb-sbert-large

This is a sentence-transformers model finetuned from NbAiLab/nb-bert-large. It builds on the previous work of the existing NbAiLab/nb-sbert-base model, using a larger foundational model and providing a larger max sequence length for inputs.

The model maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. The easiest way is to simply measure the cosine distance between two sentences. Sentences that are close to each other in meaning, will have a small cosine distance and a similarity close to 1. The model is trained in such a way that similar sentences in different languages should also be close to each other. Ideally, an English-Norwegian sentence pair should have high similarity.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: NbAiLab/nb-bert-large
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset: Subset of NbAiLab/mnli-norwegian
• Language: Norwegian and English
• License: Apache 2.0

EU AI Act

This release is a non-generative encoder model whose outputs are vectors/scores rather than language or media. Its intended functionality is limited to representation, retrieval, ranking, or classification support. On that basis, the release is preliminarily assessed as not falling within the provider obligations for GPAI models under the EU AI Act definitions, subject to legal confirmation if capability scope or marketed generality changes. For more information, see the Model Documentation Form here.

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("NbAiLab/nb-sbert-v2-large")
# Run inference
sentences = [
    "This is a Norwegian boy",
    "Dette er en norsk gutt"
    ]

embeddings = model.encode(sentences)
print(embeddings.shape)
# (2, 1024)

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9288],
#         [0.9288, 1.0000]])

Direct Usage (Transformers)

Without sentence-transformers, you can still use the model. First, you pass in your input through the transformer model, then you have to apply the right pooling-operation on top of the contextualized word embeddings.

import torch

from sklearn.metrics.pairwise import cosine_similarity
from transformers import AutoTokenizer, AutoModel

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Sentences we want sentence embeddings for
sentences = ["This is a Norwegian boy", "Dette er en norsk gutt"]

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('NbAiLab/nb-sbert-v2-large')
model = AutoModel.from_pretrained('NbAiLab/nb-sbert-v2-large')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, mean pooling.
embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
print(embeddings.shape)
# torch.Size([2, 1024])

similarity = cosine_similarity(embeddings[0].reshape(1, -1), embeddings[1].reshape(1, -1))
print(similarity)
# This should give 0.9288 in the example above.

Evaluation

Metrics

Semantic Similarity

• Dataset: STS Benchmark
• Evaluated with EmbeddingSimilarityEvaluator

MTEB (Scandinavian)