Moderato-Pro

Following intensive research into Sparse Mixture-of-Experts stability and trajectory divergence, we are pleased to introduce Moderato-V1-Pro, a flagship open-weights MoE model engineered for state-of-the-art coding, systems engineering, and agentic workflows.

Built upon 6 specialized 27B expert domains fused at the feed-forward network (FFN) layer with shared attention backbones, Moderato-V1-Pro introduces Reflexive Role Routing (RRR) — a 2-level hierarchical meta-controller that continuously evaluates hidden-state divergence and executes hot-swap expert interventions during generation without KV-cache loss.

🌟 Moderato-V1-Pro Highlights

• Reflexive Role Routing (RRR): 2-level hierarchical routing combining static token-level softmax gating G(x) with checkpointed divergence probes pθ(ht, g) at N=64 token intervals to intercept hallucinations before failure cascades occur.
• 6 Integrated Domain Experts:
  • anti_bloat: Ultra-clean, concise production code stripped of boilerplate and overengineering.
  • clean_diffs: Surgical git unified diff patches with line-level boundary preservation.
  • deep_math_cot: Formal Olympiad-level mathematical reasoning and multi-step Chain-of-Thought.
  • systems_rust: Low-level systems engineering, lock-free concurrency, and memory-safe Rust idioms.
  • modern_apis: Modern cloud/SWE architectures, asynchronous web frameworks, and schema-strict REST/gRPC APIs.
  • agentic_fable: Autonomous multi-step planning, tool orchestration, and recursive self-reflection.
• Sparse MoE Efficiency: 113.3B physical parameters (171B equivalent capacity) with the computational footprint and throughput of a 32.7B active model.
• Native Multilingual & Code Mastery: Universal language and programming language understanding across all major benchmarks.
• Flexible Thinking Control: Thinking mode with full chain-of-thought ... blocks, tuneable with reasoning_effort.

📐 Model Overview

• Model Type: Sparse Mixture-of-Experts (MoE) Causal Language Model
• Routing Mechanism: Reflexive Role Routing (RRR) Hierarchical Meta-Controller
• Total Parameters: 113.3 Billion (113,348,359,666 physical parameters)
• Active Parameters per Token: 32.7 Billion (Top-2 Experts)
• Number of Transformer Layers: 64
• Hidden Dimension (dmodel): 5120
• Feed-Forward Intermediate Dimension: 17,408
• Number of Dedicated Experts per Layer: 6 FFN Expert Pathways
• Attention Configuration:
  • Query Heads (Q): 40
  • Key/Value Heads (KV): 8 (Grouped-Query Attention - GQA)
  • Head Dimension: 128
  • Rotary Position Embedding (RoPE): Base frequency 1,000,000
• Context Length: 131,072 tokens natively, extensible up to 1,000,000 tokens via YaRN RoPE scaling.

📊 Benchmark Results

Performance Evaluation Matrix

.vl-table th{font-size:14px!important;line-height:1.2} .vl-table td:not(.benchmark-cell):not([colspan]){font-size:14px;line-height:1.2;vertical-align:middle} .vl-table .benchmark-cell{padding:10px 10px 10px 18px!important;vertical-align:middle} .vl-table .benchmark-capability{font-size:14px;font-weight:700;line-height:1.22;color:#171717} .vl-table .benchmark-name{margin-top:3px;font-size:11px;font-weight:500;line-height:1.2;color:#6B6B6B} .vl-table .metric-stack{display:flex;flex-direction:column;gap:5px;padding:2px 0} .vl-table .metric-label{font-size:10px;font-weight:500;line-height:1.1;color:#777} .vl-table .metric-value{margin-top:2px;font-size:14px;line-height:1.15;color:#171717}

🔬 Scientific Innovation: Reflexive Role Routing (RRR)

Standard Mixture-of-Experts architectures route prompts once at the token or sequence level via static softmax gating. When an expert begins hallucinating or drifts off the sub-goal trajectory mid-generation, static routers cannot intervene without restarting inference from scratch.

Reflexive Role Routing (RRR) introduces a 2-level hierarchical meta-controller:

1. Level 1 (Static MoE Gate)

Evaluates input embedding x to compute soft top-K expert weights (K=2): $$G(x) = \text{Softmax}\left(\text{TopK}(W_g x + \epsilon, k=2)\right)$$

2. Level 2 (Checkpointed Divergence Probe)

Every N = 64 tokens, a lightweight probe pθ(ht, g) analyzes the current hidden state ht against the trajectory sub-goal g, predicting divergence δ ∈ [0, 1] and confidence c ∈ [0, 1]:

• δ < 0.3: CONTINUE — proceed on the fast path.
• δ ≥ 0.3, c ≥ 0.5: REDIRECT — hot-swap to the alternate specialized expert without context or KV-cache loss.
• c < 0.5: ESCALATE — early escape to meta-orchestrator.

🚀 Quickstart & Inference

Serving with Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "nitrai-research/Moderato-V1-Pro"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)

prompt = "user\nImplement a lock-free bounded queue in Rust with atomic memory ordering.\nassistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

output = model.generate(**inputs, max_new_tokens=512, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))

High-Throughput Serving (vLLM & SGLang)

# Launch with vLLM
vllm serve nitrai-research/Moderato-V1-Pro \
    --tensor-parallel-size 4 \
    --trust-remote-code \
    --max-model-len 131072

Local Inference with llama.cpp & Ollama (GGUF)

Moderato-V1-Pro can be converted and deployed locally using llama.cpp and Ollama via GGUF quantization formats (Q4_K_M, Q5_K_M, Q8_0):

[!NOTE] How Reflexive Role Routing operates in llama.cpp / Ollama:

• Standard GGUF Execution (Level 1 MoE): In pure C++ execution engines (llama.cpp / Ollama), the model executes in high-speed native S