SlideOn

📚 Collection | 📝 Blogpost | 🛠️ Training and Evaluation

🎯 TL;DR: A 174M-parameter English sparse retrieval model built from DenseOn through sparse autoencoder (SAE) pretraining and supervised retrieval fine-tuning. SlideOn reaches 56.68 nDCG@10 on BEIR-15, surpassing DenseOn's 56.20, while producing sparse representations compatible with inverted-index search. Joint query/document Matryoshka training preserves retrieval quality under pruning, allowing search and storage costs to be adjusted at inference without retraining.

About the SlideOn / mSlideOn Family

With DenseOn and LateOn, then mDenseOn and mLateOn, we explored dense and late-interaction retrieval using our open data recipe. SlideOn and mSlideOn reuse these models' retrieval knowledge to learn sparse representations through SAE pretraining followed by supervised fine-tuning.

SlideOn starts from DenseOn for English retrieval. mSlideOn starts from mLateOn for multilingual, long-context, and code retrieval. We release both the pretrained SAE initializations and the final retrieval models.

See our blog post for the full method, results, and ablations.

SlideOn

SlideOn encodes queries and documents into 32,768-dimensional nonnegative sparse vectors. A learned linear encoder maps each normalized DenseOn token representation to its top-32 latent activations, followed by ReLU. Max pooling across tokens produces one vector per query or document, and retrieval uses the dot product between these vectors.

The dimensions are learned features, independent of the tokenizer vocabulary. Active latents can be stored in an inverted index, where query latents select posting lists and their weights contribute to document scores.

SlideOn notably:

• Reaches 56.68 on BEIR-15 with unpruned vectors, improving on DenseOn by 0.48 points.
• Reaches 56.98 with unpruned queries and documents capped at 512 nonzero latents.
• Retains 56.66 with queries capped at 64 and documents at 512, reducing the measured retrieval FLOPS proxy to 8.72.
• Supports inputs of up to 8,192 tokens and independently adjustable query and document sparsity.

Results

Headline Results (BEIR-15, nDCG@10)

BEIR-15 scores are average nDCG@10 across 15 datasets, reported on a 0–100 scale. Sparse models use dot-product scoring and exact search over each full evaluation corpus. All rows use unpruned query and document vectors. Dense and late-interaction results are taken from our DenseOn and LateOn blog post.

SlideOn achieves the highest BEIR-15 score among the sparse models below 3B parameters in this comparison. It improves on SPARSEUP by 2.22 points and on DenseOn by 0.48 points, while remaining 0.54 points behind LateOn. The figures below show how pruning changes the comparison when search and storage costs are taken into account.

Retrieval Quality versus Search and Storage Cost

Query pruning reduces the number of posting lists consulted during search. Document pruning shortens those lists and reduces the number of stored document weights. Matryoshka training concentrates retrieval information in the strongest latents so that both budgets can be adjusted after training.

src="https://cdn-uploads.huggingface.co/production/uploads/67e184d667350657a7d68aac/biV_cKSk6mQfoo0x_GSSu.png"
alt="BEIR-15 retrieval quality versus retrieval FLOPS for sparse models below 2B parameters, including SlideOn with different query pruning budgets."
style="display: block; max-width: 100%; height: auto; margin: 0 auto;"

/>

src="https://cdn-uploads.huggingface.co/production/uploads/67e184d667350657a7d68aac/kzPT8yhDHf1lIUh2Z2_7V.png"
alt="BEIR-15 retrieval quality versus average document NNZ for sparse models below 2B parameters, including SlideOn with different document pruning budgets."
style="display: block; max-width: 100%; height: auto; margin: 0 auto;"

/>

Retrieval FLOPS estimates the expected number of shared active features per query–document pair. It is a search-cost proxy, not measured latency or encoder FLOPs. Document NNZ is the average number of nonzero document features and serves as a storage-cost proxy, rather than a measurement of index size in bytes. Efficiency measurements use the fixed sample of 6,000 documents and 1,271 queries from SciFact, NFCorpus, and FiQA.

Related Checkpoints

The SAE checkpoints are starting points for training. Use SlideOn or mSlideOn for the retrieval results reported in the blog.

Training Data and Recipe

See our blog post for [training data](https://huggingface.co/blog/lighton