DenseOn-unsupervised

📚 Collection | 📝 Blog

🎯 TL;DR: The intermediate dense checkpoint produced by Stage 1 only of the DenseOn pipeline: large-scale unsupervised contrastive pre-training on filtered query-document pairs. Released as a strong starting point for your own supervised fine-tuning, knowledge distillation, or downstream adaptation.

About the LateOn / DenseOn Family

State-of-the-art retrieval is increasingly dominated by closed models, either hidden behind APIs or trained on undisclosed data. This blocks reproducibility, prevents study of possible data leakage, and gatekeeps progress to a handful of private labs. We thus decided to gather and curate a large amount of data and explore various mixtures. We release all the data used in our explorations:

• Gathered pre-training data, 1.4B query-documents pairs alongside annotations used for non-destructive filtering (structural filtering, deduplication, cross-encoder pair relevancy)
• Best pre-training mixture found with already applied filters
• Fine-tuning datasets with query, positive and 2048 mined documents alongside their scores for 1.88M samples.

Based on our findings, we trained LateOn (multi-vector/ColBERT) and DenseOn (single vector/dense) models on a proprietary Apache 2.0-compatible training dataset and release those models as well. Both are built on the ModernBERT backbone at 149M parameters, a size we believe sits at the sweet spot: large enough to handle real-world queries and documents, small enough to serve at high throughput in latency-sensitive production systems. For more information, please read our blogpost.

DenseOn-unsupervised

DenseOn-unsupervised is the output of the first stage of the DenseOn training pipeline. It has been pre-trained on a large, filtered corpus of query-document pairs using in-batch contrastive learning, but has not yet been fine-tuned with mined hard negatives.

For most production use cases, you should use the fully-trained DenseOn instead. This unsupervised checkpoint is intended for:

• Researchers studying what each pipeline stage contributes
• Practitioners who want to fine-tune on their own domain-specific data
• Distillation experiments where you want to start from a strong but un-aligned base
• Anyone running their own ablations on hard-negative mining strategies

Results

BEIR (14 datasets, NDCG@10)

DenseOn reaches 56.20 NDCG@10 on BEIR, making it the top base-size dense retriever and the first sub-150M model to clear the 56 bar. At 149M parameters it decisively beats GTE-ModernBERT (55.19) at the same size, and more tellingly outperforms snowflake-arctic-embed-l-v2.0 (55.22, 568M) and Qwen3-Embedding-0.6B (55.52, 595M) despite being roughly 4× smaller. DenseOn also stays within half a point of the strongest current-generation dense baselines, pplx-embed-v1-0.6B (56.70, 596M) and jina-embeddings-v5-text-nano (56.08, 239M), both substantially larger.

Decontaminated BEIR (12 datasets, NDCG@10)

Standard benchmark