How to run Gemma 2 with vLLM, SGLang and llama.cpp

Gemma 2 documentation is short. This page covers what the 9B instruction-tuned card and vLLM's docs state. See the Gemma 2 hub for sizes and hardware.

Engine support

• vLLM and SGLang: both are shown on the card with OpenAI-compatible servers. vLLM maps Gemma 2 to Gemma2ForCausalLM.
• llama.cpp and Ollama: the card links quantized builds (GGUF) for both.
• Transformers: update to the latest release with pip install -U transformers; the card gives no specific minimum.

Chat template

• Turns are delimited by <start_of_turn> and <end_of_turn>, with the role (user or model) after the opening delimiter.
• The card describes only user and model roles, so fold any system instructions into the first user turn.
• Use the instruction-tuned (-it) checkpoint for chat; the pre-trained one has no chat behavior.

Dtype

Weights are native bfloat16. The card notes that upcasting to float32 gives no precision gain, so serve in bfloat16.

Not stated

The card does not state a context length or rope settings, so check max_position_embeddings in the checkpoint's config before setting a length flag.