How to fine-tune Qwen3 (LoRA, QLoRA)

This guide covers adapting models from the Qwen3 family. Memory per method and GPU counts are shown below. See also LoRA, QLoRA, the training cost calculator and the fine-tuning overview.

Which checkpoint to start from

• The post-trained Qwen3 checkpoints (for example Qwen3-32B) support both thinking and non-thinking modes. Start here if you want to keep chat and reasoning behavior and adapt it.
• Qwen publishes Base checkpoints for at least some sizes (Qwen3-8B-Base appears in the Qwen organization listing on Hugging Face). Start from a Base model for domain pretraining or when you will do your own instruction tuning.

Chat template and thinking mode

Train with the template that ships with the model, which exposes enable_thinking (default true) and the /think and /no_think soft switches. The card advises leaving thinking content out of multi-turn history. If your data has no reasoning traces, format examples consistently as non-thinking turns so the adapter does not learn mixed behavior.

Adapter tips

The model card does not document LoRA target modules. Target the attention and MLP projections as you would for other decoder models, and check module names in the checkpoint before you configure the adapter. The family includes both dense and Mixture-of-Experts sizes (the Qwen3MoE architecture is named in the llama.cpp docs), so confirm which you are tuning.

Use a recent Transformers release: versions below 4.51.0 raise a KeyError on Qwen3.

License

Qwen3-32B is released under Apache 2.0, which permits commercial use and derivative works under its standard terms. Check the license field on the specific checkpoint you use.