How to fine-tune Hy4 (LoRA and full)

Tencent ships a fine-tuning pipeline with the Hy4 family. For method and cost choices see fine-tuning and the training cost calculator.

Checkpoint

Only instruct models are listed: Hy4-preview and Hy4-preview-FP8. The card mentions no base checkpoint. Train from the unquantized instruct weights.

Tooling

The finetune guide offers three routes: DeepSpeed with the Hugging Face Trainer, LLaMA-Factory and ms-swift. Full fine-tuning and LoRA are both supported. The default LoRA setup is rank 64, alpha 128, dropout 0.05.

LoRA target modules

• DeepSpeed route: q_proj, k_proj, v_proj, o_proj
• LLaMA-Factory route: q_a_proj, q_b_proj, kv_a_proj_with_mqa, kv_b_proj, o_proj
• ms-swift: set through its config

Chat template and data

Data is JSONL with message lists. The model uses the hy_v4 chat template, and reasoning_effort selects high (slow thinking) or no_think (fast). The guide says the system prompt for training and inference is empty by default, though you can customize it.

MoE note

The router's e_score_correction_bias is a buffer that the training script loads automatically. The guide says not to ignore a failure to load it.

License

The card lists Apache License 2.0, so derivatives carry no extra limits beyond that license.