mSlideOn

📚 Collection | 📝 Blogpost | 🛠️ Training and Evaluation

🎯 TL;DR: A 339M-parameter multilingual sparse retrieval model built from mLateOn through sparse autoencoder (SAE) pretraining and supervised retrieval fine-tuning. mSlideOn improves on mDenseOn by 4.76 points on MIRACL and 10.19 points on MLDR, and reaches 74.78 on MTEB Code, surpassing both mDenseOn and mLateOn. It brings multilingual, long-context, and code retrieval to sparse representations compatible with inverted-index search, with Matryoshka training for adjustable query and document sparsity.

About the SlideOn / mSlideOn Family

With DenseOn and LateOn, then mDenseOn and mLateOn, we explored dense and late-interaction retrieval using our open data recipe. SlideOn and mSlideOn reuse these models' retrieval knowledge to learn sparse representations through SAE pretraining followed by supervised fine-tuning.

SlideOn starts from DenseOn for English retrieval. mSlideOn starts from mLateOn for multilingual, long-context, and code retrieval. We release both the pretrained SAE initializations and the final retrieval models.

See our blog post for the full method, results, and ablations.

mSlideOn

mSlideOn encodes queries and documents into 32,768-dimensional nonnegative sparse vectors. Its sparse encoder lifts mLateOn's normalized 128-dimensional token representations to 768 dimensions and applies a residual MLP before selecting the top-32 latent activations and applying ReLU. Max pooling across tokens produces one vector per query or document, and retrieval uses the dot product between these vectors.

The dimensions are learned features, independent of the tokenizer vocabulary. Active latents can be stored in an inverted index, where query latents select posting lists and their weights contribute to document scores.

mSlideOn notably:

• Reaches 56.00 on BEIR-15, the highest score among the multilingual sparse models in our comparison.
• Reaches 63.21 on MIRACL target languages and 62.78 across all 18 languages, improving on mDenseOn by 4.76 points on the full benchmark.
• Reaches 75.79 on MLDR target languages and 61.68 across all 13 languages, outperforming every dense baseline in our comparison.
• Reaches 74.78 on MTEB Code, ahead of mDenseOn (71.53) and mLateOn (73.48).
• Supports inputs of up to 8,192 tokens and independently adjustable query and document sparsity.

Results

Headline Results (nDCG@10)

Scores are average nDCG@10 on a 0–100 scale. Sparse models use dot-product scoring and exact search over each full evaluation corpus. All rows use unpruned query and document vectors. Dense and late-interaction comparisons follow our mDenseOn and mLateOn blog post.

MIRACL tgt covers Arabic, English, French, German, and Spanish; MLDR tgt covers Arabic, English, French, German, Italian, Portuguese, and Spanish. These are benchmark languages included in retrieval training. Full MIRACL covers 18 languages, full MLDR covers 13, and MTEB Code averages 12 retrieval tasks.

We evaluate MIRACL using the dev split of mteb/miracl-hard-negatives and MLDR using the test split of mteb/MultiLongDocRetrieval.

Retrieval FLOPS estimates the expected number of shared active features per query–document pair. It is a search-cost proxy, not measured latency or encoder FLOPs. Measurements use the fixed sample of 6,000 documents and 1,271 queries from SciFact, NFCorpus, and FiQA.

Key Findings

Multilingual and long-document retrieval. mSlideOn improves over mDenseOn on full MIRACL and MLDR by 4.76 and 10.19 points, respectively, while remaining behind mLateOn. The per-language results show gains beyond the languages included in retrieval training.

Code retrieval. mSlideOn outperforms every sparse and late-interaction baseline in this comparison on MTEB Code.

English retrieval. mSlideOn reaches 56.00 on BEIR-15, compared with 56.70 for mDenseOn and 57.56 for mLateOn. The gains on multilingual, long-document, and code tasks therefore come with a modest reduction in English benchmark performance.

src="https://cdn-uploads.huggingface.co/production/uploads/67e184d667350657a7d68aac/fGrj4Ki_9AM0f7FjMJqyl.png"
alt="Per-dataset and per-language performance differences between mSlideOn and mDenseOn on BEIR-15, MIRACL, MLDR, and MTEB Code."
style="display: block; max-width: 100%; height: auto; margin: 0 auto;"

/>

Related Checkpoints