SlideOn-SAE

SlideOn | mSlideOn | 📚 Collection

📝 Blogpost | 🛠️ Training and Evaluation

🎯 TL;DR: The pretrained sparse autoencoder used to initialize SlideOn. It learns 32,768 latent features from frozen DenseOn token representations and provides a starting point for supervised sparse retrieval fine-tuning. Use the final SlideOn checkpoint for the retrieval results reported in our blog.

Model Details

SlideOn-SAE maps L2-normalized DenseOn token representations to their top-32 latent activations, followed by ReLU. Its decoder reconstructs the original token representations during SAE pretraining. For retrieval, the included adapter max-pools token activations into one sparse vector per query or document.

The checkpoint includes the DenseOn backbone, SAE encoder, and original SAE decoder in a single model.safetensors file. The model and tokenizer load directly from this repository.

Training Data and Recipe

See our blog post for training data, SAE pretraining, and supervised retrieval fine-tuning. Training code and the recorded training configuration are available in the code repository.

Usage

Install the supported library versions:

pip install -U "sentence-transformers>=6.1,=5.3,<6"

Load the SAE initialization and encode text into sparse vectors:

from sentence_transformers import SparseEncoder

model = SparseEncoder("lightonai/SlideOn-SAE", trust_remote_code=True)

query_embeddings = model.encode_query(["What is sparse retrieval?"])
document_embeddings = model.encode_document([
    "Sparse retrieval uses inverted indexes to search sparse vectors."
])

print(query_embeddings.shape, document_embeddings.shape)
# torch.Size([1, 32768]) torch.Size([1, 32768])

The model loaded above can also be fine-tuned with Sentence Transformers' SparseEncoderTrainer. Our code repository provides the custom retrieval loss and training scripts used for our fine-tuning recipe.

Related Checkpoints

Citation

@misc{moura2026slideon,
  author       = {Moura Junior, Paulo Roberto and Sourty, Raphaël},
  title        = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers},
  year         = {2026},
  howpublished = {Hugging Face Blog},
  url          = {https://huggingface.co/blog/lightonai/slideon-mslideon}
}