How to run Gemma 3 with vLLM, SGLang and llama.cpp

Gemma 3 comes in 1B, 4B, 12B and 27B. See the Gemma 3 hub for per-size hardware.

Engine support

• vLLM and SGLang are both listed on the 27B-it card. vLLM maps Gemma 3 to Gemma3ForCausalLM (text) and Gemma3ForConditionalGeneration (multimodal).
• llama.cpp: GGUF builds are linked from the Hugging Face launch post.
• Ollama: not covered in the sources opened, so no version is stated.
• Transformers: v4.50.0 or later per the card.

Chat template

• Use the instruction-tuned (-it) checkpoints with the built-in chat template.
• The card's examples use a system message followed by a user message, and the launch post says short system prompts work well.

Context and attention

• 128K tokens for 4B, 12B and 27B; 32K for 1B. Maximum output is 8,192 tokens.
• The architecture interleaves local sliding-window layers (1,024-token window) with global layers at a 5 to 1 ratio. The sources give no rope scaling flags.

Multimodal inputs and pitfalls

• The 1B model is text only; 4B and up accept images.
• Images are normalized to 896 x 896 and encoded as 256 tokens each. Pan and scan cropping helps non-square or high-resolution images.
• Use bfloat16: the launch post warns other dtypes may degrade quality.