Aurora1.0

Benchmarks:

piqa: acc,none: 0.6273 acc_norm,none: 0.6224

hellaswag: acc,none: 0.2922 acc_norm,none: 0.3220

arc_easy: acc,none: 0.4987 acc_norm,none: 0.4491

arc_challenge: acc,none: 0.2099 acc_norm,none: 0.2500

arithmark_3.0: acc,none: 0.3390 acc_norm,none: 0.3390

Int Index based on OpenSLM-Leaderboard: 15.09 It matches or slightly beats GPT2-Small on most Benchmarks

Architecture:

Context Lenght: 1024

Factorized & Tied Embeddings Vocabulary Size: 32768 Embedding Rank: 384

GQA Attention (2:1 Head ratio) 10 Query and 5 Key/Value heads Head Dimension: 64 Positional Encoding: RoPE (θ=10000) QK-Norm

30 identical Transformer Layers Hidden Size: 640 Intermediate Size: 1728 Activation: SwiGLU

Training:

Muon and AdamW Around 150M Parameters 7 Billion tokens of Pretraining 11 Hours on an RTX Pro 6000

Quick Start

You can use the Inference.py script.

Example command: inference.py --Prompt "The capital of Denmark is" --temperature 0.1 --max_new_tokens 32

Output: The capital of Denmark is Copenhagen, which has a population of about 1.5 million people. The city is located in the northern part of the country and is surrounded by the sea.