ettin-encoder

🎯 TL;DR: State-of-the-art paired encoder and decoder models (17M-1B params) trained identically for fair comparison with open data. Encoders beat ModernBERT. Decoders beat Llama 3.2/SmolLM2.

📄 Paper | 🚀 GitHub Repository

This model is part of the Ettin suite - the first collection of paired encoder-only and decoder-only models trained with identical data, architecture, and training recipes. Ettin enables fair comparisons between encoder and decoder architectures across multiple scales, providing state-of-the-art performance for open-data models in their respective size categories.

Table of Contents

• Performance Highlights
• Quick Start
• Model Description
• Training Data
• Model Family
  • Encoder Models
  • Decoder Models
  • Cross-Objective Models
• Accessing Training Checkpoints
• Research Applications
• Training Details
• Model Architecture
• Usage Examples
• Fine-tuning Examples
• Citation

📊 Performance Highlights

Encoder Tasks (vs. ModernBERT)

• GLUE Average: 88.9 vs 88.4 (Base), 90.8 vs 90.4 (Large)
• MTEB v2 English Retrieval: 45.7 vs 43.9 (Base), 48.4 vs 47.0 (Large)
• Code Search and Long Context: Superior performance on CodeSearchNet and MLDR

Decoder Tasks (vs. SmolLM2 & Llama 3.2)

• Average Score: 46.2 vs 45.2 (SmolLM2-135M)
• 1B Model: 59.0 vs 56.6 (Llama 3.2-1B)
• Generative Tasks: Competitive across all model sizes

Key Finding

Architecture-specific advantages persist: A 400M encoder outperforms a 1B decoder on classification tasks, while a 400M decoder outperforms a 1B encoder on generation tasks.

🚀 Quick Start

Installation

pip install torch>=1.9.0
# until the new pip release, install from main to use decoders (transformers>=4.54.X will contain it)
# encoders work with transformers>=4.48.0
pip install git+https://github.com/huggingface/transformers.git

30-Second Examples

Encoder for Classification/Embeddings:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-encoder-150m")
model = AutoModel.from_pretrained("jhu-clsp/ettin-encoder-150m")

Decoder for Text Generation:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-decoder-150m")
model = AutoModelForCausalLM.from_pretrained("jhu-clsp/ettin-decoder-150m")

Model Description

Ettin models are designed to provide a foundation for comparing encoder-only and decoder-only architectures. Unlike previous comparisons that were limited by different training data, architectures, and recipes, Ettin models use:

1. Identical training data - Same high-quality mixture across all models
2. Open Training Data - Data is available now with batch-level training data for each of the 250+ checkpoints
3. Matched architectures - Only differing in attention patterns (bidirectional vs causal) and training objectives (MLM vs CLM)
4. Consistent training recipe - Three-phase training with 2T tokens
5. Multiple scales - From 17M to 1B parameters

This approach allows for true apples-to-apples comparisons between encoder and decoder models, revealing the inherent strengths of each architecture.

Training Data

The training data is publicly available and split across different phases:

• Pre-training Data: jhu-clsp/ettin-pretraining-data - 1.7T tokens of diverse data mixture
• Mid-training/Extension Data: jhu-clsp/ettin-extension-data - 250B tokens of higher-quality filtered data
• Decay Phase Data: jhu-clsp/ettin-decay-data - 100B tokens of premium data sources
• Training Data Order: jhu-clsp/ettin-data-order - Batch-level training order (columns: input_ids, step)

Model Family

Encoder Models

Decoder Models