How to run Granite 4.0 with vLLM and Transformers

This guide covers serving the Granite 4.0 family. Memory sizing and launch commands for each size appear below.

Architecture and engines

The H Small card describes a hybrid decoder-only design: 4 attention layers and 36 Mamba2 layers, with a mixture-of-experts of 72 experts and 10 active per token (32B total, 9B active). The family also includes Micro Dense (3B), H Micro Dense (3B), H Tiny MoE (7B) and H Small MoE (32B).

• The card shows Transformers (AutoModelForCausalLM, device_map="auto") and vllm serve, which exposes an OpenAI-compatible API on port 8000.
• The card does not state minimum engine versions, and it does not document SGLang, llama.cpp or Ollama usage. Check each engine's own release notes first.

Chat template and tools

• Turns use role tokens such as <|start_of_role|> and <|end_of_role|>.
• A default system prompt guiding toward professional, accurate and safe answers was added in October 2025.
• Tool calling follows the OpenAI function schema. The model emits JSON inside <tool_call> tags.
• Multilingual quality outside English may need few-shot examples. Supported languages: English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch and Chinese.

Long context

The sequence length is 128K tokens. The base card says the H Small model uses NoPE (no position embeddings) rather than RoPE, unlike the Micro Dense variant, so there is no YaRN or rope scaling setting documented for it. Context length is limited by memory, not a scaling flag.

Pitfalls

• Pick the right checkpoint: the instruct models are the chat ones, the -base models are for completion and Fill-in-the-Middle code tasks.
• Weights are Apache 2.0.