How to run Mistral Small 3.2 with vLLM

This guide covers serving the Mistral Small family, based on the Mistral-Small-3.2-24B-Instruct-2506 card. Sizing and launch commands per model appear below.

Engine support

• vLLM is the recommended engine. The card says to use vLLM 0.9.1 or newer, which should install mistral_common 1.6.2 or newer.
• Transformers works with mistral-common 1.6.2 or newer, using MistralTokenizer and Mistral3ForConditionalGeneration.
• The card does not mention llama.cpp, Ollama or SGLang.

Serving flags

The card's vLLM command uses the Mistral-native formats:

• --tokenizer_mode mistral
• --config_format mistral
• --load_format mistral
• --tool-call-parser mistral with --enable-auto-tool-choice for function calling
• --limit-mm-per-prompt to allow images (the card sets 10 per prompt)

Prompting

• Use a low temperature. The card recommends 0.15.
• Load the system prompt from SYSTEM_PROMPT.txt in the model repository, applying the date formatting shown in the usage examples.
• The model accepts image inputs. The card's examples use a maximum of 131072 tokens.

Pitfalls

• Use all three mistral format flags together, as in the card's command.
• Tool calling needs both the parser flag and auto tool choice.