Best open chat models

For a chat assistant you want an instruct-tuned model that follows instructions, holds a conversation, and answers fast. Pick on license, context length, and size, then test it on your own prompts. This page covers how to choose; the catalog below lists the instruct models.

What matters when picking a chat model

Instruct, not base. Base checkpoints continue text and do not reliably follow instructions. Chat apps want the instruct (or "it", "chat") variant from the same publisher.

Latency and concurrency. Chat is interactive, so time to first token and decode speed shape the experience. Smaller models, and mixture-of-experts models that run only some weights per token, respond faster. If many users share one GPU, the KV cache for each conversation also eats memory.

Context length. Long chats and document questions need room. Longer context costs memory, so match it to what your product really sends.

Tool calling and structured output. If the assistant calls functions or returns JSON, check that the card advertises native support.

License. Apache 2.0 and MIT cards carry no extra use restrictions of their own. Some chat models use custom community licenses with conditions, so read the card.

Precision. BF16 is the reference; a quantized build shrinks the footprint. Use the VRAM calculator to size it.

Picks by situation

Smallest and cheapest to serve: Phi-4-mini-instruct. Microsoft's card lists a 128K context and an MIT license, and the catalog shows about 3.8B parameters. It is a good fit for high-volume, low-latency assistants where a small model is enough.

Smaller GPUs (a 24 GB card): Mistral-Small-24B-Instruct-2501. The card lists 24B parameters, a 32k context window, and Apache 2.0, and highlights native function calling and JSON output. A 24B model in BF16 is a stretch for 24 GB, so expect to run a quantized build there or step up a card size.

One 80 GB GPU: gpt-oss-120b. OpenAI's card lists 117B parameters with 5.1B active and Apache 2.0, and says it fits a single 80GB GPU. Only a fraction of the weights run per token, which helps latency. It also has configurable reasoning effort, so you can keep chat replies quick or let it think longer on hard questions.

Lower latency from the same family: gpt-oss-20b. 21B parameters with 3.6B active, Apache 2.0, aimed at lower-latency and local use per the card.

Longest context: Phi-4-mini-instruct at 128K is the longest among these picks. For multimodal chat with even longer windows, see the vision-language page.

Most permissive license: Mistral Small and the gpt-oss models are Apache 2.0, and Phi-4-mini-instruct is MIT, all per their cards.

Serving notes

Run the model behind an inference server that batches requests, since chat traffic is bursty. Keep the system prompt short, because it is re-read on every turn. For a first deployment, pick the smallest model that passes your own evaluation set, then scale up only if it fails.

Aquanode rents GPUs by the hour, so you can try a 4B model and a 100B-class model on the right card sizes and keep the one that works. See pricing. Model-card claims are vendor-reported, so measure on your own traffic.