mSlideOn-SAE

SlideOn | mSlideOn | 📚 Collection

📝 Blogpost | 🛠️ Training and Evaluation

🎯 TL;DR: The pretrained sparse autoencoder used to initialize mSlideOn. It learns 32,768 latent features from frozen mLateOn token representations and provides a starting point for supervised multilingual and code retrieval fine-tuning. Use the final mSlideOn checkpoint for the retrieval results reported in our blog.

Model Details

mSlideOn-SAE lifts L2-normalized, 128-dimensional mLateOn token representations to 768 dimensions and applies a residual MLP before projecting to 32,768 latent features, selecting the top-32 activations, and applying ReLU. Its linear decoder reconstructs the original 128-dimensional token representations during SAE pretraining. For retrieval, the included adapter max-pools token activations into one sparse vector per query or document.

The checkpoint includes the mLateOn backbone and token projections, SAE encoder, and original SAE decoder in a single model.safetensors file. The model and tokenizer load directly from this repository.

Training Data and Recipe

See our blog post for training data, SAE pretraining, and supervised retrieval fine-tuning. Training code and the recorded training configuration are available in the code repository.

Usage

Install the supported library versions:

pip install -U "sentence-transformers>=6.1,=5.3,<6"

Load the SAE initialization and encode text into sparse vectors:

from sentence_transformers import SparseEncoder

model = SparseEncoder("lightonai/mSlideOn-SAE", trust_remote_code=True)

query_embeddings = model.encode_query([
    "What is sparse retrieval?",
    "Qu’est-ce que la recherche éparse ?",
])
document_embeddings = model.encode_document([
    "Sparse retrieval uses inverted indexes to search sparse vectors.",
    "La recherche éparse utilise des index inversés pour rechercher des vecteurs épars.",
])

print(query_embeddings.shape, document_embeddings.shape)
# torch.Size([2, 32768]) torch.Size([2, 32768])

encode_query() and encode_document() insert the [Q] and [D] markers automatically. Pass raw text to both methods.

The model loaded above can also be fine-tuned with Sentence Transformers' SparseEncoderTrainer. Our code repository provides the custom retrieval loss and training scripts used for our fine-tuning recipe.

Related Checkpoints

Citation

@misc{moura2026slideon,
  author       = {Moura Junior, Paulo Roberto and Sourty, Raphaël},
  title        = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers},
  year         = {2026},
  howpublished = {Hugging Face Blog},
  url          = {https://huggingface.co/blog/lightonai/slideon-mslideon}
}