Mistral-Small-3.2-2506

[!NOTE] Includes our GGUF chat template fixes! Tool calling works as well! If you are using llama.cpp, use --jinja to enable the system prompt.

Run in llama.cpp:

./llama.cpp/llama-cli -hf unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:UD-Q4_K_XL --jinja --temp 0.15 --top-k -1 --top-p 1.00 -ngl 99

Run in Ollama:

ollama run hf.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:UD-Q4_K_XL

• Temperature of: 0.15
• Set top_p to: 1.00
• Max tokens (context length): 128K

Mistral-Small-3.2-24B-Instruct-2506

Mistral-Small-3.2-24B-Instruct-2506 is a minor update of Mistral-Small-3.1-24B-Instruct-2503.

Small-3.2 improves in the following categories:

• Instruction following: Small-3.2 is better at following precise instructions
• Repetition errors: Small-3.2 produces less infinite generations or repetitive answers
• Function calling: Small-3.2's function calling template is more robust (see here and examples)

In all other categories Small-3.2 should match or slightly improve compared to Mistral-Small-3.1-24B-Instruct-2503.

Key Features

• same as Mistral-Small-3.1-24B-Instruct-2503

Benchmark Results

We compare Mistral-Small-3.2-24B to Mistral-Small-3.1-24B-Instruct-2503. For more comparison against other models of similar size, please check Mistral-Small-3.1's Benchmarks'

Text

Instruction Following / Chat / Tone

Infinite Generations

Small 3.2 reduces infitine generations by 2x on challenging, long and repetitive prompts.

STEM

Vision

Usage

The model can be used with the following frameworks;

• vllm (recommended): See here
• transformers: See here

Note 1: We recommend using a relatively low temperature, such as temperature=0.15.

Note 2: Make sure to add a system prompt to the model to best tailer it for your needs. If you want to use the model as a general assistant, we recommend to use the one provided in the SYSTEM_PROMPT.txt file.

vLLM (recommended)

We recommend using this model with vLLM.

Installation

Make sure to install vLLM >= 0.9.1:

pip install vllm --upgrade

Doing so should automatically install mistral_common >= 1.6.2.

To check:

python -c "import mistral_common; print(mistral_common.__version__)"

You can also make use of a ready-to-go docker image or on the docker hub.

Serve

We recommand that you use Mistral-Small-3.2-24B-Instruct-2506 in a server/client setting.

1. Spin up a server:

vllm serve mistralai/Mistral-Small-3.2-24B-Instruct-2506 --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice --limit_mm_per_prompt 'image=10' --tensor-parallel-size 2

Note: Running Mistral-Small-3.2-24B-Instruct-2506 on GPU requires ~55 GB of GPU RAM in bf16 or fp16.

2. To ping the client you can use a simple Python snippet. See the following examples.

Vision reasoning

Take leverage of the vision capabilities of Mistral-Small-3.2-24B-Instruct-2506 to take the best choice given a scenario, go catch them all !

from datetime import datetime, timedelta

from openai import OpenAI
from huggingface_hub import hf_hub_download