ERNIE-5.5-Flash

Introduction

ERNIE 5.5 Flash is an efficient sparse Mixture-of-Experts (MoE) language model with 280B total parameters and 14B activated per token, pre-trained from scratch on 33.5T tokens with a training context of up to 128K; fine-tuning and inference support a 1M-token context. Different domains, reasoning tasks and agentic applications call for different data, training objectives and behavioral policies; by releasing the weights at the end of pre-training, we let developers build on the model's general capabilities and choose for themselves where to take it next. Because further training and agentic use keep adding to compute and memory cost, we made efficiency a key design goal from the start, revisiting each component, from attention and residual connections to the MoE layers and the output head, and treating activated compute, KV cache and long-context cost as part of one overall design. The same long-term view shaped the data and the training system: pre-training data emphasizes capabilities that are hard to build in afterwards, such as repository-scale code and long-context understanding, and the training system was co-designed with the architecture so that its efficiency choices also train efficiently at scale.

ERNIE-5.5-Flash-Base is the checkpoint at the end of pre-training, before any mid- or post-training, and can serve as a common starting point for domain adaptation, instruction tuning and reinforcement-learning research. Under a unified evaluation pipeline it delivers strong results across knowledge, reasoning, math, code, multilingual and long-context benchmarks; full results are in the table below, reported alongside DeepSeek-V4-Flash-Base (284B-A13B), Hy3-preview-Base (295B-A21B) and MiMo-V2.5-Base (310B-A15B), all evaluated with the same pipeline, prompts and few-shot settings.

For more details on ERNIE 5.5 Flash Base, please refer to our technical blog.

Model Downloads

ERNIE 5.5 Flash Highlights

Architecture

1. 280B-A14B hybrid-attention MoE. 43 layers (1 dense FFN + 42 Multi-Router Latent MoE) with 37 Heavily Compressed Attention (HCA) layers and 6 Multi-head Latent Attention layers with sparse attention (MLA + DSA); 512 routed experts (top-10) plus 1 shared expert per MoE layer; pre-trained from scratch on 33.5T tokens with the Muon optimizer and a training context of up to 128K; fine-tuning and inference support 1M context.

2. Hybrid MLA + HCA attention with layer-specific head dimensions. One in seven layers keeps full-context access through MLA; the rest use Heavily Compressed Attention (HCA), which caches a single entry per 128 tokens. Head dimensions are matched to each layer's attention cost: the cheap HCA layers, which attend over a 128×-compressed sequence, use wide 512-dim heads, while the more compute-intensive token-level MLA layers use narrower 256-dim heads. At a comparable KV-cache budget, pairing HCA with MLA outperforms pairing it with Compressed Sparse Attention (CSA). The MLA layers adopt DSA, matching dense attention in pre-training loss while their per-query core attention cost no longer grows with context length.

3. Low-Rank Gated Attention and Virtual Head Attention (VHA). The attention output gate is computed from the low-rank query latent instead of the hidden state, cutting gate-projection parameters by 75% while improving performance. VHA adds a lightweight low-rank post-mix across heads with an identity skip, fused into the attention operator at inference. Both are applied in every attention layer, together with QK-norm.

4. Multi-Router Latent MoE: multiple routers over half-width latent experts. A single router's scores correlate only weakly with expert utility, especially in the middle-to-deep layers. Multi-Router replaces it with several independently initialized sigmoid routers with distinct preferences, so expert selection is no longer dictated by any single router's errors and becomes more accurate, while the number of activated experts, expert FLOPs and communication volume stay unchanged. Each MoE layer has 512 routed experts (top-10) plus 1 shared expert, operating in a 2048-d latent space so that expert dispatch/combine happens at half the hidden width. Auxiliary-loss-free load balancing keeps expert load even; following the Stable LatentMoE design, an RMSNorm before the up-projection and SiTU-GLU expert activations keep expert outputs well-scaled and stabilize optimization; on top of this RMSNorm, each routed expert's routing weight is multiplied by a learnable scalar of its own.

5. ECHO: the backbone's own final layer replaces the MTP module. Instead of a separate MTP module, ECHO reuses the backbone's final layer to predict future tokens, so the multi-horizon loss directly shapes the next-token parameters; the same layer doubles as a built-in draft model for speculative decoding.

6. Manifold-constrained Hyper-Connections (mHC). The model adopts mHC with expansion rate 4 in place of the standard residual stream, strengthening signal propagation across the 43 layers while preserving expressivity.

7. MultiMax Output Head. A MultiM