Instella-MoE-Think

Instella-MoE is a state-of-the-art fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters trained end-to-end from pre-training to RL. Trained from scratch on AMD Instinct™ MI300X and MI325X GPUs using AMD's Primus framework, Instella-MoE combines a sparsely activated MoE design with architectural innovations such as Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective.

Figure 1: Pre-trained and Post-trained Instella-MoE model performance compared with other similar size state-of-the-art models.

Takeaways

• Instella-MoE is a new state-of-the-art fully open Mixture-of-Experts language model developed by AMD, with 16 billion total parameters and 2.8 billion active parameters per token, trained from scratch on AMD Instinct™ MI300X and MI325X GPUs.
• The Instella-MoE model checkpoint release spans every stage of the model training pipeline, including pre-training, mid-training, long-context extension, SFT, DPO and RL.
• Built entirely on the AMD ROCm™ software stack on top of the Primus training and Miles RL frameworks, Instella-MoE incorporates cutting-edge architecture and systems innovations—including Gated Multi-head Latent Attention (Gated MLA) and extreme communication-computation overlap through FarSkip-Collective—for efficient large-scale training and inference on AMD hardware.
• Fully open and accessible: we provide our complete training recipe across all training stages, including training frameworks, data mixtures, intermediate checkpoints and inference code.

The release includes checkpoints from the following Instella-MoE training pipeline stages as shown in Table 1 below:

Model Summary

Results

Pretraining Results

Table 2: Instella-MoE-16B-A3B-Base Results on Standard Benchmarks.

Table 3: Instella-MoE-16B-A3B-Base Results on Long Context HELMET and RULER Benchmarks.

Post-training Results

Table 4: Instella-MoE-Think results. We evaluate all models using the OLMES framework, generating up to a maximum of 32768 tokens.

Getting Started

Example Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "amd/Instella-MoE-16B-A3B-Think"

tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", trust_remote_code=True)

prompt = [{"role": "user", "content": "What are the computational benefits of Mixture-of-Experts models?"}]
inputs = tokenizer.apply_chat_template(
    prompt,
    add_generation_prompt=True,
    return_tensors='pt'
)

tokens = model.generate(
    inputs.to(model.device),
    max_new_tokens=1024,
    temperature=0.6,
    top_p=0.95,
    do_sample=True
)

print(tokenizer.decode(tokens[0], skip_special_tokens=False))

For high-throughput inference with SGLang, please refer to the setup and usage instructions in our GitHub repository.

Training Details

Instella-MoE was trained end-to-end on AMD Instinct™ MI300X and MI325X GPUs using the AMD ROCm™ software stack, built on top of the Primus training framework and the Miles RL framework. Training proceeds through a multi-stage pipeline—pre-training, mid-training, long-context extension, SFT, DPO, and RL—with each stage progressively strengthening the model's capabilities.

For the complete training recipe, including per-stage data mixtures, hyperparameters, training frameworks, and inference code, please refer to our GitHub repository and technical blog.

Acknowledgements

We are deeply grateful to the LLM360 team and the Miles team for their invaluable support throughout the development of our model.

License

• The Instella-MoE models are licensed for academic and research purposes under a ResearchRAIL license.
• Refer to the LICENSE fo