How to run Hy4 with vLLM and SGLang

This guide covers serving the Hy4 family. The checkpoint is named Hy4-preview, with an FP8 build (FP8). Memory sizing and launch commands appear below.

Engine support

The card documents vLLM and SGLang only, each through a prebuilt image: vllm/vllm-openai:hy4-preview and lmsysorg/sglang:hy4-preview (the latter for x86 and ARM). It states no minimum version, so use those images rather than a stock release. It does not cover llama.cpp or Ollama.

vLLM flags

• --tensor-parallel-size 8
• --speculative-config with method mtp and 3 speculative tokens
• --attention-backend FLASHMLA_SPARSE
• --tool-call-parser hy_v4
• --reasoning-parser hy_v4

SGLang flags

• --tp-size 8
• --reasoning-parser auto
• --speculative-algorithm NEXTN with --speculative-num-steps 3

Reasoning toggle

Reasoning defaults to high. For direct answers pass chat_template_kwargs with reasoning_effort set to no_think through extra_body.

Context and sampling

Context is 1M tokens. The card documents no rope or YaRN override. It recommends temperature=0.9 and top_p=1.0.

Pitfalls

• The vLLM example sets a sparse attention backend explicitly.
• Parser names are hy_v4, not a generic reasoning parser.
• The card does not say whether remote code is required, so follow the image's own example command.