How to run Llama 3.2 with vLLM and llama.cpp

This guide covers serving the Llama 3.2 family: small text models (1B) and vision models (11B Vision shown here).

Engine support

• The 11B Vision card requires Transformers 4.45.0 or newer for conversational inference with images, and says it is compatible with vLLM.
• The 1B card lists Transformers 4.43.0 or newer, and notes quantized versions through llama.cpp, Ollama, LM Studio and Jan (see GGUF).

Vision input

Pass images inside chat messages and format them with processor.apply_chat_template() using add_generation_prompt=True.

Tool calling

vLLM documents the llama3_json parser for all Llama 3.1 and 3.2 models. Use the 3.2 template (tool_chat_template_llama3.2_json.jinja), which adds image support. vLLM says parallel tool calls are not supported for Llama 3.

Context and languages

Both cards list a 128k token context. The 1B card names English, German, French, Italian, Portuguese, Hindi, Spanish and Thai as officially supported.

EU restriction on multimodal use

The vision card states that the Section 1(a) rights are not granted to individuals domiciled in, or companies with a principal place of business in, the European Union for the multimodal models. Check this before deploying the vision sizes.