Speck2

Speck2-140M is a 140.7M parameter English base language model that interleaves global grouped-query attention with gated causal convolution. It was pretrained from scratch on 20B tokens using a three-phase curriculum that shifts from broad web text toward math, synthetic, and stylistically diverse high-quality text.

This is a base model, not instruction-tuned or specialized in any way. It has no chat template and no safety alignment.

Summary

Architecture

18 residual blocks: 8 global attention + 10 gated causal convolution, each followed by a SwiGLU feed-forward.

Input/output embeddings (640-wide) are tied and connect to the 768-wide residual stream via learned projections.

Usage

Speck2-140M works with the Transformers Auto classes through its bundled custom model and tokenizer code. Set trust_remote_code=True when loading it.

pip install "transformers==5.1.0" torch sentencepiece safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "specklabs/Speck2-140M"
device = "cuda" if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
).to(device)

prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

output = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=False,
)
generated = output[0, inputs.input_ids.shape[1] :]
print(tokenizer.decode(generated, skip_special_tokens=True))

The bundled generation path is validated for single-prompt greedy decoding. Direct forward passes support right-padded batches when use_cache=False.

Training

Muon optimized 2D matrix parameters; AdamW (beta 0.9/0.95, epsilon 1e-8) handled embeddings, norms, and conv kernels. The run used a single RTX 5090 rented through Vast.ai.

Training documents were globally deduplicated after NFKC, lowercase, and whitespace normalization. The curriculum used three mixture phases:

Evaluation

The quality columns combine the Open SLM Leaderboard at revision 2eafcfc647b667e67f3b0288e9b67da497a78052 and BananaMind Base Bench 1.1 at revision d4aade51312889e8580963e1ce960c6eaef1a450. No chat template or generation was used for the four Speck evaluations.

Benchmarks and speed

Open SLM Int Index means the chance-normalized Intelligence Index reported by the Open SLM Leaderboard. BananaMind Base Bench 1.1 Elo means the overall Elo reported by BananaMind Base Bench 1.1. Speed and memory values are local batch-1 measurements described below. Reference models were pretrained on 1.5 to 100 times as many tokens. This is a parameter-adjacent comparison, not a compute-matched one.

Inference speed

Speed was measured locally at batch 1 with eager PyTorch, model-native caches, last-token logits, and tokenization excluded. Prefill uses 512 tokens. Decode measures 64 greedy cached steps after a 448-token prefix and includes argmax. CPU runs use FP32 with 16 threads; RTX 3090 runs use BF16. Reported th