How to run Nemotron 3 Nano with vLLM and SGLang

This guide covers serving the Nemotron 3 family, using the Nemotron-3-Nano-30B-A3B card as the reference.

Architecture

The card describes a Mamba2-Transformer hybrid mixture-of-experts with 52 layers: 23 MoE layers, 23 Mamba-2 layers and 6 grouped-query attention layers.

Engine support

• vLLM 0.12.0 or newer. The card says it needs a custom reasoning parser file (nano_v3_reasoning_parser.py) supplied with the model.
• SGLang: launch with --reasoning-parser nano_v3.
• TRT-LLM: --reasoning_parser nano-v3 with the autodeploy backend.
• Transformers: the chat template is integrated from v5.3.0.
• The card only points to a model browser for llama.cpp quantizations and says nothing about Ollama, so check those engines yourself.

Reasoning toggle

Reasoning is on by default. Pass enable_thinking=False to apply_chat_template() to turn the trace off. The card suggests temperature 1.0 and top_p 1.0 for reasoning, and temperature 0.6 with top_p 0.95 for tool calling.

Long context

The model supports up to 1M tokens, but the default Hugging Face config is 256K because of memory. For 1M in vLLM, set VLLM_ALLOW_LONG_MAX_MODEL_LEN=1. No YaRN or rope scaling flags are documented.

Pitfalls

• trust_remote_code is required because of the custom architecture.
• Without the matching reasoning parser, thinking text appears inline in the reply.