Vela-1.0-Omni-Nano

Text, images, speech and environmental audio in one embedding space. Vela Omni Nano supports multimodal search, routing and clustering with normalized vectors that can be compared directly.

Try Vela Studio · Vela collection · Detailed evaluation

Model overview

A frozen CLAP audio branch adds environmental-sound information to the existing speech representation. Text and image computations are retained; audio embeddings are newly trained and evaluated. Architecture and measured identity.

Evaluation

Scores are 0–100; higher is better. The comparison uses the same held-out evaluation examples, complete retrieval pools and 128-token text cap for both models. Bold marks improvement over the original small model. These known test pools are reused across releases.

The common protocol uses labeled TRAIN prototypes for text classification and all matching positives for retrieval; it is separate from official MTEB classification. All 14 metrics, Macro-F1, exact counts and uncertainty.

Standard benchmarks and rankings

The primary metric, Mean(TaskType), weights each task type equally. Mean(Task) weights individual tasks equally and is supplementary. Rankings use complete benchmark results from the September 17, 2026 registry snapshots plus both current Vela models for English/audio and Nano for multilingual; “≤ size” counts models with no more total parameters than this model.

Multilingual coverage remains a limitation. All 131 task scores and protocol.

Nano ranks 5/75 on the English panel and 6/27 on the audio panel among models with no more than its 163.8M total parameters. These are snapshot-relative comparisons across reported protocols, including single-modality specialists. The original small has not been evaluated on these complete panels. Full rankings and both aggregate metrics · All task scores and methods.

Audio Mean(TaskType) improves from 46.97 to 52.34 over the previous Nano. Parameters rise 20.97%; speech–text retrieval and VehicleSoundClustering regress. Complete gains and trade-offs.

Quality and model size

Each plot combines the full benchmark score, global and size-constrained ranks, and the gap to the best model at no greater total size. Every complete model with a known size is plotted, including models below the observed Pareto frontier. Highlighting Vela does not imply frontier membership. Click either figure for full resolution. The English panel uses Nano's default shared text and Mini's optional official-instruction mode; audio uses the default mode for both.

Selected task strengths

These task-level comparisons highlight specific strengths; they do not establish overall benchmark SOTA.

Comparison data and methodology · Figure gallery

Usage

Use PyTorch and matching torchaudio, Transformers 4.57.6, Hugging Face Hub, safetensors, NumPy, and Pillow:

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("llm-semantic-router/Vela-1.0-Omni-Nano")
sys.path.insert(0, path)
from vela_omni import VelaOmni

model = VelaOmni.from_pretrained(path, device="cpu")
vectors = model.encode_text(["A bicycle beside a building.", "Someone is reading aloud."])
print(vectors.shape)  # (2, 384)

Text inputs support up to 512 tokens, including special tokens. Longer inputs raise ValueError; shorten or explicitly chunk them.

Pass a list of Pillow images to model.encode_image(images). Pass original NumPy waveforms to model.encode_audio(waveforms, sampling_rate=48000) using their actual 16,000, 44,100 or 48,000 Hz rate; each waveform must be at most 30 seconds. Mono or channels-first arrays are supported. Keep the original waveform: the speech and CLAP branches independently derive their 16 kHz and 48 kHz inputs. Do not downsample to 16 kHz before calling the API when higher-rate PCM is available. Compare normalized vectors with their dot product. To route media, embed each destination’s name and description, then select the closest vector. To discover routing categories, cluster media embeddings and inspect each group. Similarity scores are rankings, not calibrated probabilities.

Training and license

Cross-modal alignment uses COCO image–caption pairs from the CC-BY 2.0