Ming-3.0-audio-tiny

Ming-3.0-audio-tiny is a compact native audio-language model from inclusionAI. It accepts audio together with text instructions and supports speech recognition, spoken question answering, audio understanding, audio reasoning, and optional synthesized-speech responses.

Model Summary

Highlights

• Efficient audio intelligence: Provides speech and audio understanding in a compact Mixture-of-Experts model.
• Complex speech recognition: Handles Chinese and English speech, dialects, accents, noisy environments, and multi-speaker recordings.
• Long-audio understanding: Comprehends and analyzes long-form recordings.
• Multidimensional perception: Understands speech, non-speech sounds, acoustic scenes, and music.
• Audio reasoning: Answers questions and performs reasoning over audio content.
• Audio interaction: Produces text responses and can optionally synthesize speech with built-in voices or a reference voice.

Architecture

Ming-3.0-audio-tiny combines an audio encoder and projector with a compact Mixture-of-Experts language backbone. An optional Talker generates speech responses from the model's output.

Quick Start with Transformers

The model uses custom Transformers code. Review the repository code before enabling trust_remote_code=True.

git clone https://github.com/inclusionAI/Ming-3.0-audio-flash.git
cd Ming-3.0-audio-flash

python -m venv .venv-transformers
source .venv-transformers/bin/activate
pip install -r requirements-transformers.txt

Audio Understanding

python examples/transformers_audio.py examples/audio/introduce_ant_group.wav \
  --model inclusionAI/Ming-3.0-audio-tiny \
  --prompt "Please answer the question in the audio."

Speech Recognition

python examples/transformers_audio.py examples/audio/introduce_ant_group.wav \
  --model inclusionAI/Ming-3.0-audio-tiny \
  --task asr \
  --max-new-tokens 256 \
  --no-repeat-ngram-size 10

The validated ASR prompt asks the model to identify the spoken language and place a tab between the language label and transcription.

Direct Python Usage

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "inclusionAI/Ming-3.0-audio-tiny"
audio_path = "examples/audio/introduce_ant_group.wav"

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
    use_fast=False,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
).eval()

messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": audio_path},
            {"type": "text", "text": "Please answer the question in the audio."},
        ],
    }
]

inputs = processor(messages).to(model.device)
inputs["audio_feats"] = inputs["audio_feats"].to(model.dtype)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=False,
    )

generated_ids = output_ids[0, inputs["input_ids"].shape[1] :]
response = processor.decode(generated_ids, skip_special_tokens=True).strip()
print(response)

Speech Output

The Talker example supports built-in voices and reference-audio voice prompting:

python examples/transformers_thinker_talker.py \
  --model inclusionAI/Ming-3.0-audio-tiny \
  --audio examples/audio/introduce_ant_group.wav \
  --voice US_Stella \
  --output output.wav

See the voice guide for available voice names, invocation rules, and reference-audio usage.

Serving

vLLM-Omni

Text output or ASR uses the Tiny Thinker on one GPU:

export MODEL_PATH=inclusionAI/Ming-3.0-audio-tiny
MING_MODEL_VARIANT=tiny MING_OUTPUT_MODE=text \
  ./scripts/serve_vllm_omni.sh "$MODEL_PATH"

For synthesized-speech output, select the full-audio profile:

export MODEL_PATH=inclusionAI/Ming-3.0-audio-tiny
MING_MODEL_VARIANT=tiny MING_OUTPUT_MODE=audio \
  ./scripts/serve_vllm_omni.sh "$MODEL_PATH"

The reference full-audio profile places the Thinker on GPU 0 and the Talker on GPU 1. Select the deployment YAML that matches text-only, ASR, or Talker output. See the deployment guide for details.

Select an IP or registered voice with the vLLM-Omni client's --voice option:

python examples/vllm_omni_audio_chat.py examples/audio/introduce_ant_group.wav \
  --audio-response \
  --voice US_Stella \
  --output-audio outputs/answer.wav

The client sends this selection as Chat Completions audio.voice: {"id": "..."}. vLLM-Omni uses the built-in IP voice 灵小柔 when no voice is specified. The model directory also provides 灵楷 and US_James as preset reference voices under voice_presets/voice_audio/, with their matching transcripts in voice_presets/voice_prompts.json. Both names are also available as built-in IP voices.

SGLang

export MODEL_PATH=inclusionAI/Ming-3.0-audio-tiny
MING_MODEL_VARIANT=tiny CUDA_VISIBLE_DEVICES=0 \
  ./scripts/serve_sglang.sh "$MODEL_PATH"

The validated Tiny SGLang profile uses one GPU and provides text output. See the SGLang example for client usage.

SGLang-Omni

SGLang-Omni serves transcription, audio chat, and speech synthesis from one server. The T