BananaMind-2-Pro

BananaMind-2-Pro is a 138,971,520-parameter decoder-only base language model trained from scratch by BananaMind. The completed pretraining run processed 99,999,449,088 tokens—the scheduled realization of a 100B-token curriculum—over 184,954 optimizer steps.

The model has a 3,072-token context window and a custom 32,768-token digit-aware byte-level BPE tokenizer. Its architecture uses grouped-query attention, QK normalization, RoPE, SwiGLU, RMSNorm, tied input/output embeddings, and a KV cache for generation.

This is a base model, not an instruction-tuned or chat model. Use continuation-style prompts and load the repository with trust_remote_code=True.

Release Status

Model Details

Architecture Overview

The model maps tokens through a tied input embedding, 24 pre-normalized Transformer blocks, a final RMSNorm, and the tied language-model head. Each block contains causal grouped-query self-attention followed by a SwiGLU feed-forward network, with residual connections around both sublayers.

Eight query heads share four key/value heads, reducing KV-cache size while retaining multi-head query capacity. QK normalization stabilizes attention logits, and RoPE supplies position information without learned absolute-position embeddings. During autoregressive generation, each layer can reuse cached keys and values instead of recomputing the full prefix.

Evaluation

These results describe the exact final checkpoint at step 184,953. ARC Easy, ARC Challenge, PIQA, and HellaSwag use zero-shot acc_norm,none. ArithMark 3 uses length-normalized continuation accuracy, while ArithMark 2 uses raw continuation accuracy. Code Elo is the Base Bench 1.1 code-completion category; Base Bench 1.1 Elo is the complete 350-item suite.

The INT Index chance-normalizes HellaSwag, the mean of ARC Easy and ARC Challenge, PIQA, and ArithMark 3. ArithMark 2 is not included:

N(s, c) = 100 * (s - c) / (100 - c)
INT = [N(H,25) + N((ARC_E+ARC_C)/2,25) + N(P,50) + 0.65*N(A3,25)] / 3.65

INT Index vs Training Compute

Training compute is estimated consistently as 6 × parameters × training tokens. Using the exact 138,971,520-parameter count gives 83,382.91 PFLOPs for Pro Final at 100B tokens and 43,279.49 PFLOPs for Pro Preview at 51.904512B tokens. GPT-X2.5 uses 135M parameters, 75B tokens, and a 2,048-token context window; context length is not a separate term in this 6ND approximation.

Base Bench Checkpoint Progression

This progression contains 41 complete 350-item evaluations from 2.70B tokens through the final 99.999B-token checkpoint. Every point uses CUDA, bfloat16, and batch size 1 for consistency. The final batch-1 endpoint is 1132 Elo, 236/350 correct (67.43%), and 64.69% weighted accuracy. The primary comparison table above retains the separately measured batch-32 result.

Base Bench 1.1 Category Results

The final model passed 233 of 350 cases for 66.57% raw accuracy and 63.73% weighted accuracy. Its overall fixed-item rating is 1124 Elo.

Base Bench was evaluated locally with CUDA, bfloat16, and batch size 32. ArithMark 3 used CUDA bfloat16 with batch size 64; ArithMark 2 used CUDA bfloat16 with batch size 1. Resul