sentence-bert-swedish-cased

This is a sentence-transformers model: It maps Swedish sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. This model is a bilingual Swedish-English model trained according to instructions in the paper Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation and the documentation accompanying its companion python package. We have used the strongest available pretrained English Bi-Encoder (all-mpnet-base-v2) as a teacher model, and the pretrained Swedish KB-BERT as the student model.

A more detailed description of the model can be found in an article we published on the KBLab blog here and for the updated model here.

Update: We have released updated versions of the model since the initial release. The original model described in the blog post is v1.0. The current version is v2.0. The newer versions are trained on longer paragraphs, and have a longer max sequence length. v2.0 is trained with a stronger teacher model and is the current default.

Usage (Sentence-Transformers)

Using this model becomes easy when you have sentence-transformers installed:

pip install -U sentence-transformers

Then you can use the model like this:

from sentence_transformers import SentenceTransformer
sentences = ["Det här är en exempelmening", "Varje exempel blir konverterad"]

model = SentenceTransformer('KBLab/sentence-bert-swedish-cased')
embeddings = model.encode(sentences)
print(embeddings)

Loading an older model version (Sentence-Transformers)

Currently, the easiest way to load an older model version is to clone the model repository and load it from disk. For example, to clone the v1.0 model:

git clone --depth 1 --branch v1.0 https://huggingface.co/KBLab/sentence-bert-swedish-cased

Then you can load the model by pointing to the local folder where you cloned the model:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("path_to_model_folder/sentence-bert-swedish-cased")

Usage (HuggingFace Transformers)

Without sentence-transformers, you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.

from transformers import AutoTokenizer, AutoModel
import torch

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Sentences we want sentence embeddings for
sentences = ['Det här är en exempelmening', 'Varje exempel blir konverterad']

# Load model from HuggingFace Hub
# To load an older version, e.g. v1.0, add the argument revision="v1.0"
tokenizer = AutoTokenizer.from_pretrained('KBLab/sentence-bert-swedish-cased')
model = AutoModel.from_pretrained('KBLab/sentence-bert-swedish-cased')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, max pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)

Loading an older model (Hugginfface Transformers)

To load an older model specify the version tag with the revision arg. For example, to load the v1.0 model, use the following code:

AutoTokenizer.from_pretrained('KBLab/sentence-bert-swedish-cased', revision="v1.0")
AutoModel.from_pretrained('KBLab/sentence-bert-swedish-cased', revision="v1.0")

Evaluation Results

The model was evaluated on SweParaphrase v1.0 and SweParaphrase v2.0. This test set is part of SuperLim -- a Swedish evaluation suite for natural langage understanding tasks. We calculated Pearson and Spearman correlation between predicted model similarity scores and the human similarity score labels. Results from SweParaphrase v1.0 are displayed below.

The following code snippet can be used to reproduce the above results:

from sentence_transformers import SentenceTransformer
import pandas as pd

df = pd.read_csv(
    "sweparaphrase-dev-165.csv",
    sep="\t",
    header=None,
    names=[
        "original_id",
        "source",
        "type",
        "sentence_swe1",
        "sentence_swe2",
        "score",