Serving LLMs with vLLM: A Practical Inference Guide

Most vLLM tutorials start at pip install vllm and end at a working curl request. That leaves you stuck the first time the model runs out of memory at a context length that looked fine, or throughput lands at a fraction of what you expected. Those problems are easier to reason about once you know what the engine does with your prompt.

So this guide goes the long way round: neural networks and transformers, the full inference path, attention and the KV cache with a worked example, what is inside a model download, and only then vLLM's features and parallelism. It finishes with a hands-on deployment on a rented GPU.

TL;DR: vLLM is an open-source inference engine that serves LLMs behind an OpenAI-compatible API. Its core trick is PagedAttention, which manages the KV cache in small fixed-size blocks so many requests can share GPU memory without waste, combined with continuous batching so new requests join the running batch instead of waiting. To use it well you need to understand three things: how prefill and decode differ, why the KV cache dominates memory at long context, and which attention backend your model's positional encoding allows. Scale beyond one GPU with --tensor-parallel-size.

Neural network and transformer foundations

A short refresher first, because every vLLM setting you will touch later maps back to one of these ideas.

What is a neural network?

A neural network is a program built from stacked layers of small computing units called neurons. Each neuron multiplies its inputs by learned values, adds them up, and passes the result through a simple function. Stacked in layers, they build up structure: early layers of a language model pick up surface patterns like which word pieces go together, and later layers combine those into abstract features like grammar, topic, and intent.

Physically, a network is a large collection of numeric arrays plus the code that says how to combine them. Loading a model onto a GPU means copying those arrays into GPU memory so the chip can run the matrix multiplications in parallel. When a model is too large for one GPU, the arrays are split across several GPUs, sometimes several machines, which exchange intermediate results over fast links such as NVLink. PyTorch handles the plumbing, and inference engines like vLLM decide how to split the work.

During inference, data flows forward through the layers once per step: the forward pass. During training, the network also runs a backward pass that compares its output to the correct answer and nudges every weight. Inference needs only the forward pass, so it uses far less memory: no gradients and no optimizer state.

Embeddings, weights, and quantization

These three terms come up constantly, so it helps to pin them down.

Embeddings. Text is split into tokens, and each token becomes a vector of learned numbers called an embedding, often thousands of dimensions long. What the model stores is the embedding matrix, one row per vocabulary token. The embeddings for your prompt are looked up fresh from that matrix on every request; they are never saved.

Weights. Weights are every learned parameter, including the embedding matrix and each layer's matrices, typically stored as 32-bit or 16-bit floats. A model file is essentially weights plus a little metadata. No prompts or training examples live in it as text.

Quantization. Quantization stores weights at lower precision, for example 8-bit integers, 4-bit integers, or 8-bit floats (FP8) instead of 16-bit floats. Fewer bits means less memory and less data to move per token, usually faster generation, at the cost of some precision and possibly a small drop in output quality. Careful 8-bit quantization is often hard to tell apart from the original; aggressive 4-bit can degrade harder tasks.

vLLM serves quantized checkpoints in formats such as GPTQ, AWQ, INT8, INT4, and FP8. Prefer offline quantization: convert once, save the checkpoint, and point vLLM at it. On-load quantization works for some formats but slows every startup. Better still, use a pre-quantized version from the model's authors or a reputable publisher, and if you do convert, do it on a GPU.

What is a transformer, and why is it so effective?

A transformer is a neural network architecture built for sequences such as sentences, code, or chat histories. Nearly every modern LLM is one.

Its defining feature is attention: when processing a token, the model can look at every other token in the input and weigh how relevant each one is. Older sequence models read strictly left to right and carried a compressed summary forward, so early information faded. A transformer has direct access to the whole context at every step. That is what lets it answer a question about something mentioned twenty turns ago or follow an instruction buried in a long system prompt. It is also why inference gets expensive as context grows.

The LLM inference workflow

Here is what happens between sending a prompt and getting an answer back. Every inference engine, vLLM included, follows this shape.

Step-by-step workflow

1. Load the model. Once, at startup, the engine reads the architecture definition and weights from disk (or downloads them) into GPU memory. This is why a large model can take minutes to become ready.
2. Receive the input. A prompt: a single line or a full chat history with a system message.
3. Tokenize. The tokenizer splits the text into tokens (words, word fragments, punctuation, control symbols) and maps each to an integer ID using the model's vocabulary files.
4. Look up embeddings. Each token ID indexes a row of the embedding matrix, giving one vector per token.
5. Prefill. All prompt tokens run through the layers in one parallel, compute-heavy pass. Each attention layer calculates a key and a value vector for every prompt token and saves them in the KV cache, which is what makes the next phase cheap.
6. Decode. The answer is generated one token at a time. Each step processes only the new position and reads everything about earlier tokens from the KV cache. Each step is light on compute but must stream the weights out of memory, so decode speed is usually limited by memory bandwidth.
7. Apply sampling parameters. Each decode step scores every vocabulary token. Temperature sharpens or flattens the distribution, top-k limits the choice to the k most likely tokens, and top-p to the smallest set whose combined probability reaches p. Together they set how creative or conservative the output is.
8. Detokenize. The chosen token IDs are mapped back to text.
9. Return the output, whole or as a stream.

Steps 6 through 8 repeat until the model emits an end-of-sequence token, hits a stop string you specified, or reaches the maximum number of new tokens.

Metrics that matter when you benchmark

When you measure an inference setup, four families of numbers matter:

• Latency: how fast one request feels. Time to first token (TTFT) is mostly prefill; the time between later tokens is decode.
• Throughput: total work done, usually output tokens per second across all requests, or requests per second.
• Concurrency: requests in flight at once. Throughput rises with concurrency until the GPU saturates while per-request latency worsens, so always state the concurrency a number was measured at.
• Memory usage: GPU memory taken by weights, KV cache, and overhead, which caps how much context and concurrency fit.

A benchmark is only meaningful for the workload shape it was run on.

Additional concepts you will meet in inference

A few more terms show up in model cards, vLLM flags, and error messages.

Context window

The context window is the maximum number of tokens the model handles at once: system prompt, chat history, retrieved documents, the question, and generated output. Past the limit, older content is truncated or summarized, or the request is rejected. A bigger window enables long chats and retrieval over many documents, but every token needs KV cache space and lengthens prefill. In vLLM, --max-model-len sets the context limit the server will accept; lowering it below the model's maximum is a common way to free KV cache memory for more concurrent requests.

Vocabulary and tokenizers

The vocabulary is the full set of tokens a tokenizer can produce. Sizes vary widely: some models use around 32,000 tokens, others well over 150,000. A larger vocabulary can represent some languages and scripts more compactly, so the same sentence may take fewer tokens, which lowers latency and cost for that text.

Tokenizers also differ in algorithm. Byte-pair encoding (BPE), SentencePiece, and tiktoken-style tokenizers all split text differently, so the same paragraph can produce noticeably different token counts on different models. Two consequences follow. First, compare token-based prices and speeds with care across model families. Second, always use the exact tokenizer that ships with the model. Mixing a tokenizer from one variant with weights from another is a quiet way to get garbage output.

Positional encoding

Attention on its own has no sense of order. Without extra information, "the dog bit the man" and "the man bit the dog" contain the same tokens and would look the same to the model. Positional encoding fixes this by injecting information about where each token sits in the sequence.

Different models use different schemes, and two matter most in practice:

• ALiBi (Attention with Linear Biases) does not add anything to the embeddings. Instead, it adds a penalty to attention scores that grows with the distance between two tokens, so nearby tokens naturally count for more. BLOOM uses ALiBi.
• RoPE (Rotary Position Embedding) rotates the query and key vectors by an angle that depends on each token's position, so the relative distance between tokens is encoded directly in their dot product. LLaMA, Qwen, Mistral, and most current open models use RoPE, and RoPE can be scaled after training to stretch the context window further.

Here is the practical point. The positional scheme is fixed when the model is trained; it is part of the architecture. The attention backend, the low-level kernel that computes attention, is chosen when you serve the model, and it has to support that scheme. Some fast kernels are built around RoPE and do not implement ALiBi's score biases, so they simply cannot run an ALiBi model correctly. More general implementations like PyTorch's SDPA handle both. We will come back to this in the attention backend section.

Positions apply to every token, prompt and generated alike, and they are computed at runtime for whatever index each token lands on.

CUDA graphs

Every GPU kernel launch has a small CPU-side cost, and each decode step runs many small kernels, so launch overhead adds up. CUDA graphs record a sequence of GPU operations once and replay it with a single launch, a natural fit since decoding repeats the same computation with different data. The graph changes how work is dispatched, not what the model computes.

vLLM captures graphs for common batch sizes at startup, which is part of why startup is slow. When debugging a startup failure or short on memory, --enforce-eager disables capture at some cost in decode speed.

Warmup

The first few requests to a freshly started server are often slower than the rest: memory pools are still being allocated, some kernels are compiled on first use, and caches are cold. Benchmarking without a handful of warmup requests first blends that one-time cost into what should be a steady-state measurement, so send a few throwaway requests before you start the clock.

Worked example: from prompt to answer

Let's make this concrete. We will send the prompt "how are you?" and follow how a model arrives at "I am fine." First with a deliberately tiny network, to show what a layer does, then with the attention and KV cache mechanics that real LLMs rely on.

A tiny network, layer by layer

A basic feed-forward network has three kinds of layer:

• An input layer that receives the data. For a language model, this is where token IDs become embedding vectors.
• One or more hidden layers that transform their input using learned weights, a bias, and an activation function.
• An output layer that produces the prediction, here a score for the next token.

For illustration, imagine a toy network with four input positions (one per prompt token), four hidden neurons, and a single output neuron. Real models are enormously larger, but the arithmetic has the same form.

Tokenization. "how are you?" becomes four tokens: how, are, you, ?. Each is mapped to its integer ID.

Input layer. Each ID selects a row of the embedding matrix. Suppose the first token's vector is [0.1, 0.2, 0.3, 0.4]. The input layer's job is only to hand these vectors onward; the representations that carry real meaning are produced in the layers after it.

Hidden layer. Each hidden neuron takes a weighted sum of its inputs, adds its own bias, and passes the total through an activation function such as ReLU or tanh. For the first hidden neuron that looks like:

h1 = activation(w1_1 * 0.1 + w1_2 * 0.2 + w1_3 * 0.3 + w1_4 * 0.4 + b1)

The other three hidden neurons do the same with their own weights and biases, giving h1 through h4.

Output layer. The output neuron combines the hidden values the same way:

output = activation(v1 * h1 + v2 * h2 + v3 * h3 + v4 * h4 + b_out)

In a real LLM the output layer produces one score for every token in the vocabulary, not a single number. Those scores say how likely each token is to come next.

Prediction. The model picks the highest-scoring token, "I", or samples among the top candidates depending on your sampling settings.

Repeat. The generated token is appended to the input, which is now how, are, you, ?, I, and the process runs again to predict "am", then "fine", then an end-of-sequence token that stops generation.

Real models have dozens of layers with thousands of dimensions and attention layers interleaved, but the principle holds. Notice the waste, though: the naive version reprocesses "how are you?" at every step. That is the problem the KV cache solves.

The same prompt, with attention and the KV cache

Now let's follow the same request through a transformer with a KV cache, the way vLLM actually runs it.

Tokenization and embedding. As before, the prompt becomes four token IDs, and each ID indexes a row in the embedding matrix to produce a vector.

Prefill: building the cache. The model runs all four prompt tokens through its layers at once. Inside every attention layer, each token's vector is multiplied by three learned weight matrices to produce three new vectors:

• a query, which represents what this token is looking for,
• a key, which represents what this token offers to others,
• a value, which is the information this token will contribute if another token pays attention to it.

Each attention layer is split into several heads, and each head has its own set of these weight matrices, so different heads can learn to track different kinds of relationship (one might focus on grammar, another on which noun a pronoun refers to).

Within a layer, each prompt token compares its query against the keys of the tokens it is allowed to see (in a decoder model, itself and everything before it) using a dot product. The raw scores go through a softmax, which turns them into weights that are all positive and add up to one. The token's new representation is the weighted blend of those tokens' value vectors. That is self-attention: every token gets to pull in exactly as much information from each earlier token as it has learned is useful.

The crucial side effect: the key and value vectors for all four prompt tokens, in every layer, are written to the KV cache. After prefill, the cache holds the full context of "how are you?" and no output token has been produced yet.

Decode: using the cache. Now the model generates, and for each new token the loop looks like this:

1. Query. Compute a query vector for the current position only.
2. Attention scores. Take the dot product of that query with every key stored in the cache.
3. Softmax. Normalize the scores into attention weights.
4. Weighted sum. Blend the cached value vectors using those weights. This mixes in context from the prompt and from every token generated so far.
5. Project to logits. After passing through the remaining layers, the final representation is multiplied by the output projection to produce a logit, a raw score, for every token in the vocabulary.
6. Choose. Pick the highest logit (argmax, also called greedy decoding) or sample from the probability distribution using temperature, top-k, and top-p.
7. Detokenize. Convert the chosen ID back to text. Here the first result is "I".
8. Extend the cache. Compute the key and value for "I" and append them to the cache, then repeat for the next position.

On the next step, the query for the position after "I" is compared against five cached keys (the four prompt tokens plus "I"), producing "am". Then six, producing "fine". Then seven, producing the stop token.

Compare that with the naive table. With the cache, the prompt's keys and values are calculated exactly once, during prefill. Each decode step computes only one new query, key, and value per layer, and everything else is a read from memory. This is why decode is fast per step but bottlenecked by memory bandwidth, and why the KV cache grows with every token of context and output. On a long conversation with many concurrent users, the cache can take more GPU memory than the model weights themselves.

This token-by-token loop, where each output depends on the ones before it, is called autoregressive decoding. GPT-style models, LLaMA, Qwen, Mistral, BLOOM, and Falcon all generate this way. Models like BERT, which read the whole input bidirectionally and fill in masked positions, do not decode token by token and do not use a KV cache for generation. If you are unsure what kind of model you have, the config.json fields architectures, is_decoder, is_encoder_decoder, and use_cache usually tell you, which brings us to what else is in a model download.

What attention is, and how to choose an attention backend

In plain terms: when you answer a question about a long report, you do not only recall the last paragraph; you pull up the parts that bear on the question and weigh them more. Attention does the same for a model. For each new token, it looks back over the whole context and decides how much each earlier piece matters right now.

A few clarifications:

• Attention works over the context window only. Nothing carries over between separate requests unless the engine deliberately caches it (vLLM's prefix caching does that for repeated prefixes).
• The KV cache is an inference-time optimization, not part of the model.
• An attention backend is the GPU kernel that performs the attention math. It is a serving-time choice, but you cannot change the model's attention design or positional scheme; you can only pick a backend that implements it correctly.

Attention backends compared

vLLM ships several backends and chooses one automatically based on your GPU, the model, and the data type. You can override the choice (historically through the VLLM_ATTENTION_BACKEND environment variable; check the docs for your vLLM version), which is worth knowing when a model misbehaves or crashes with one kernel. The main options:

• Torch SDPA: PyTorch's built-in scaled dot-product attention. It is the broad-compatibility baseline and works with both ALiBi and RoPE models. It is rarely the fastest, but it is the safe fallback when you are unsure or another kernel is unstable.
• FlashAttention (v2 and v3): highly optimized kernels that avoid writing the full attention matrix to GPU memory. They are an excellent match for RoPE models such as Qwen, LLaMA, and Mistral. They require supported head dimensions and data types, v3 targets Hopper-generation GPUs like the H100 and H200, and they are not the right choice for ALiBi models.
• Triton attention: kernels written in OpenAI's Triton language. A reasonable choice for ALiBi models when you want more speed than SDPA offers.
• FlashInfer and others: specialized high-performance kernels whose availability depends on how your vLLM build was compiled and what hardware you run. Check the support matrix for your device before relying on one.

The rule of thumb: identify the model's positional encoding from its config or model card, rule out any backend that does not support it, then pick the fastest of what remains for your GPU generation.

Model architecture and artifacts

When you download a model from Hugging Face, you get a folder of files, not a single blob. Knowing what each one does helps when something fails to load.

• Weights. The learned parameters, usually split across several .safetensors files (older repos may use .bin). This is the bulk of the download, from a few gigabytes for small models to hundreds for the largest.
• config.json. The architecture's hyperparameters: number of layers, hidden size, number of attention heads, vocabulary size, maximum position embeddings, and the positional encoding strategy. vLLM reads this to know how to build the model.
• Tokenizer files. Typically tokenizer.json, tokenizer_config.json, and depending on the tokenizer type, vocabulary and merge files plus a list of special tokens. The chat template, if the model has one, usually lives in tokenizer_config.json.
• generation_config.json (optional). The authors' default generation settings, such as temperature, top-p, and end-of-sequence token IDs. Hugging Face Transformers applies them unless you override them, which keeps behavior consistent and close to what the authors intended.
• Adapters (optional). Small sets of extra weights that specialize a base model without retraining it. LoRA (Low-Rank Adaptation) is the most common method, part of the broader PEFT (Parameter-Efficient Fine-Tuning) family. An adapter is typically megabytes against a base model of gigabytes, and the community shares them for tasks like chat style, code, or domain knowledge. Load one on top of the base model to change its behavior; vLLM can serve several at once.
• Custom code (rare). A few models need Python code from their own repository to load, enabled with trust_remote_code. That code runs with your process's permissions, so only enable it for sources you trust, and pin the revision so it cannot change underneath you.

How model artifacts load into CPU and GPU memory

When an inference framework loads a model, the pieces go to different places:

1. The config is read into CPU (system) memory. It is small and only describes the architecture.
2. The tokenizer files are loaded into CPU memory too. Tokenization runs on the CPU.
3. The weights, including the embedding matrix, are loaded into GPU memory, where all the heavy math happens.

Keeping these consistent matters: the tokenizer's IDs have to line up with the rows of the embedding matrix, and the layer shapes in the weights have to match what the config describes. That is why you should always pull the tokenizer and weights from the same repository and revision.

You can inspect a model's config and tokenizer without loading any weights at all. This is a quick sanity check to run on a freshly rented GPU box before you commit to a multi-hour download of the full checkpoint. On an Aquanode pod, SSH in and run:

python3 -m venv ~/venvs/hf
source ~/venvs/hf/bin/activate
pip install --upgrade pip transformers sentencepiece

Then save and run this inspection script:

from transformers import AutoConfig, AutoTokenizer

model_id = "bigscience/bloom"

config = AutoConfig.from_pretrained(model_id)
print("Architecture:", getattr(config, "architectures", "N/A"))
print("Layers:", getattr(config, "num_hidden_layers", getattr(config, "n_layer", "N/A")))
print("Hidden size:", getattr(config, "hidden_size", "N/A"))
print("Vocab size:", getattr(config, "vocab_size", "N/A"))

tokenizer = AutoTokenizer.from_pretrained(model_id)
text = "Write a short poem about the moon."
ids = tokenizer.encode(text, add_special_tokens=True)
print("Token IDs:", ids)
print("Token count:", len(ids))
print("Decoded:", tokenizer.decode(ids))

It downloads only the small config and tokenizer files, so it finishes in seconds even for a 176-billion-parameter model, and it uses no GPU at all. Files cache under ~/.cache/huggingface; on a rented box, point HF_HOME at persistent storage if you have it so the eventual weight download survives a restart. Gated models need HF_TOKEN set.

The operational takeaways: token counts (not characters) drive latency and cost, tokenizers are model-specific so measure with the one you will serve, and tokenizer and weights must come from the same revision.

Model licenses: what to check before you deploy

A model being downloadable does not mean you can use it however you like. Before you put a model behind an endpoint, especially a commercial one, read its license and check these points:

• License type. Standard open-source (Apache 2.0, MIT), research-only or non-commercial, or a custom license with its own conditions? Many popular models ship under custom community licenses with restrictions a standard open-source license would not have.
• Commercial use. Is it allowed at all, and does it require registering, requesting approval, or accepting extra terms above some user count or revenue level?
• Redistribution and derivatives. Can you share the weights, fine-tuned versions, merged adapters, or quantized copies? Quantizing or fine-tuning often counts as creating a derivative with its own obligations.
• Attribution and acceptable use. Some licenses require specific attribution wording, and some (the RAIL family, for example) forbid certain applications and require you to pass those restrictions on to your own users.

Keep a copy of the license with anything you distribute, record the exact model and revision you deployed, and get legal advice when a term is ambiguous.

Model profiles: BLOOM-176B vs Qwen-72B

Two older but instructive open models show how the concepts above turn into real deployment decisions. Always treat the official model card as the authoritative source for specs; the notes below summarize what each card states and what follows from it. The memory figures are computed from the published parameter counts at 2 bytes per parameter for BF16/FP16.

BLOOM-176B (bigscience/bloom)

• Size and memory. 176 billion parameters. At 2 bytes per parameter, the BF16/FP16 weights alone come to roughly 350GB, before any KV cache or runtime overhead. That is more than four 80GB GPUs combined, so serving it at full precision means eight 80GB cards such as the H100 or A100 80GB, a set of higher-memory cards like the H200, multiple nodes, or a quantized checkpoint.
• Context and positions. Trained with a short context of around 2,000 tokens, using ALiBi positional biases. Per the ALiBi discussion above, that rules out FlashAttention, and Torch SDPA or Triton attention are the backends to use.
• Tokenizer and prompts. It uses a Hugging Face fast tokenizer with a very large multilingual vocabulary. It is a base model with no built-in chat template, so you have to supply your own prompt format if you want chat-style interaction.
• License. BigScience BLOOM RAIL 1.0, an open license with use-based restrictions.
• Read the model card.

Qwen-72B (Qwen/Qwen-72B)

• Size and memory. 72 billion parameters. Per its model card, BF16/FP16 needs roughly 144GB of total GPU memory, and the INT4 variant fits in roughly 48GB. In practice the full-precision version wants at least two 80GB GPUs (tight once you add KV cache) or two H200s for comfortable headroom, while the INT4 version can run on a single 80GB card.
• Context and positions. Supports a 32k-token context using an extended form of RoPE. FlashAttention v2 is supported, and SDPA is the safe fallback.
• Tokenizer and prompts. A tiktoken-based tokenizer with a vocabulary above 150,000 tokens. Loading it needs trust_remote_code, so make sure your runtime's Transformers version matches what the repository expects. This is the base model, so no chat template is needed; the chat variants ship their own.
• License. The Tongyi Qianwen license, which allows commercial use under its specific terms.
• Read the model card.

Put side by side, the two show why "which model" and "which GPU" are the same question. The larger model is not just more expensive to host; its ALiBi positions and short context also narrow your backend choices and cap the tasks it suits. The smaller one fits on less hardware, handles far longer inputs, and runs on the fastest attention kernels. For the sizing math on any other model, run it through the VRAM calculator or see how much VRAM you need for LLMs.

vLLM core concepts and features

With the foundations in place, vLLM is easy to describe: an open-source library for serving LLMs efficiently, originally from UC Berkeley. It loads models straight from Hugging Face, streams output, and exposes an OpenAI-compatible API, so existing client code only needs a new base URL. It runs on NVIDIA, AMD, and other accelerators, across multiple GPUs and machines.

Its main features, and what each one buys you:

• PagedAttention. The feature vLLM is known for. Instead of reserving one big contiguous chunk of memory for each request's KV cache sized to the maximum possible length, it splits the cache into small fixed-size blocks and allocates them on demand, much like an operating system pages virtual memory. Memory that would sit reserved-but-unused under a naive scheme becomes available for other requests, so more requests fit in the same GPU at once. Blocks can also be shared between requests with a common prefix.
• Continuous batching. A static batch waits until every request in it finishes before starting the next batch, so one long answer holds up many short ones. vLLM schedules at the level of individual decode steps: finished requests leave the batch and new ones join immediately. This keeps the GPU full and is a large part of vLLM's throughput advantage over naive serving.
• Optimized attention backends. FlashAttention, FlashInfer, Triton, and SDPA, selected automatically or overridden as discussed above.
• CUDA and HIP graphs. Captured at startup to cut kernel launch overhead during decode.
• Quantization support. GPTQ, AWQ, INT4, INT8, FP8, and more, so you can fit larger models on smaller GPUs or serve more requests per card. FP8 in particular benefits from Hopper-generation hardware.
• Speculative decoding. A small, fast draft model (or a similar mechanism) proposes several tokens ahead, and the main model verifies them in one pass. When the guesses are right, you get several tokens for the price of one decode step. How much this helps depends heavily on the workload.
• Chunked prefill. Long prompts are split into pieces and interleaved with decode steps from other requests, so one huge prompt does not stall everyone else's generation.
• Prefix caching. When many requests share the same beginning, such as a long system prompt or the same document in a RAG pipeline, vLLM can reuse the already-computed KV blocks for that prefix instead of prefilling it again. Enabled with --enable-prefix-caching in versions where it is not on by default.
• Multi-LoRA serving. One base model can serve many LoRA adapters at once, with each request naming the adapter it wants. This is far cheaper than running a separate full model per fine-tune.
• Multimodal models. Vision-language and other multimodal architectures are supported alongside text-only ones.
• Streaming and an OpenAI-compatible API. Endpoints like /v1/chat/completions and /v1/models, so OpenAI SDKs work out of the box.
• Metrics. A Prometheus endpoint exposing queue depth, KV cache usage, throughput, and latency.

Parallelism in vLLM

Once a model or its traffic outgrows one GPU, you need to split the work. There are four ways to do that, and vLLM's support for each is different.

Tensor parallelism (TP). Each layer's big matrix multiplications are sliced across GPUs, so every GPU does part of the work for every layer and they exchange partial results after each step. This is the primary way vLLM scales a model too large for one card, and it is what most deployments use. You set it with:

--tensor-parallel-size 4

The usual choice is the number of GPUs in the node. TP communicates constantly, so it works best between GPUs linked by NVLink within one machine; across slower PCIe links it still works but the communication overhead eats more of the gain. The number of attention heads generally needs to divide evenly by the TP size, which is why 2, 4, and 8 are the common values.

Pipeline (model) parallelism. Different layers live on different GPUs or nodes, and activations are passed from one stage to the next. Training frameworks such as Megatron-LM and DeepSpeed build heavily on this, with sophisticated scheduling. vLLM does offer a --pipeline-parallel-size option, mainly for spreading a model across multiple nodes, but it does not implement pipeline parallelism in the same comprehensive way those frameworks do, and for single-node serving tensor parallelism is the standard approach. A common multi-node pattern combines the two: tensor parallelism within each node, pipeline parallelism across nodes.

Data parallelism. Each GPU (or GPU group) holds a full model copy and serves different requests. The simple way is several independent vLLM servers behind a load balancer; newer releases also have built-in options. This scales throughput, not the model size you can fit.

Expert parallelism. Mixture-of-Experts (MoE) models route each token to a few of many expert sub-networks. Expert parallelism places different experts on different GPUs; it is model-specific and more involved to tune, so check vLLM's parallelism docs for current flags.

How vLLM handles a request, end to end

Putting everything together, a vLLM server works like this:

1. Startup. Load the tokenizer and weights, split the weights across the tensor-parallel ranks, measure how much GPU memory is left, and set aside KV cache blocks from that remainder (the share of total GPU memory vLLM may claim is governed by --gpu-memory-utilization). Capture CUDA graphs.
2. Serve. Start the OpenAI-compatible HTTP server on the configured host and port.
3. Per request. Tokenize, hand the request to the scheduler, which batches it with other in-flight requests, run prefill and then decode steps, and detokenize.
4. Respond. Stream tokens back as they are produced, or return the whole text at the end, updating the KV cache as each new token is generated and freeing its blocks when the request finishes.

Behind that loop, vLLM handles the scheduling, memory management, and GPU-to-GPU communication for whichever parallelism you configured. Your job is mainly to pick the right model, precision, context length, and parallelism for your hardware, and to use a vLLM build or container image that matches your accelerator and driver.

Deploy vLLM on an Aquanode GPU

Here is the hands-on part: getting a vLLM server running on a rented GPU and answering requests.

1. Size the model, then rent the GPU. Work out the memory first: weights at your chosen precision plus room for the KV cache at the context length and concurrency you need. The VRAM calculator does this for common models. As rough guides, a 7-8B model in BF16 runs comfortably on a single 24-48GB card like the L40S, a 70B-class model needs two or more 80GB GPUs at BF16 or one 80GB card at INT4, and anything larger means a multi-GPU node. Then pick a machine on the Aquanode marketplace or launch one from pods. Live per-hour rates across providers are on the marketplace and on the GPU index, so check there rather than trusting any number in a blog post.

2. SSH in and confirm the GPUs are visible.

nvidia-smi

You should see every GPU you rented, with its memory and driver version.

3. Start the vLLM server. The official vllm/vllm-openai Docker image bundles vLLM and its dependencies, which avoids most CUDA and PyTorch version conflicts:

docker run --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN=$HF_TOKEN \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

What each part does:

• The -v mount keeps downloaded weights on the host so a restart does not re-download them. HF_TOKEN is only needed for gated models.
• --ipc=host gives the container the shared memory tensor parallelism relies on.
• --model is the Hugging Face repository ID; --max-model-len caps context and so the KV cache per sequence; --gpu-memory-utilization is the share of each GPU vLLM may claim.

For multiple GPUs, add --tensor-parallel-size 2 (or your GPU count). For a quantized checkpoint, point --model at the quantized repository. The first start is slow while weights download and CUDA graphs are captured; wait for the log line saying the server is listening on port 8000.

4. Verify the server. From the same machine:

curl http://localhost:8000/v1/models

This should return JSON listing the model you loaded. Then send a real request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "how are you?"}],
    "max_tokens": 64
  }'

5. Reach it from your laptop. The simplest private option is an SSH tunnel, which forwards the port without exposing the server to the internet:

ssh -L 8000:localhost:8000 <user>@<your-pod-address>

Now http://localhost:8000/v1 on your laptop points at the server, and any OpenAI SDK client works by setting that as its base URL. If you do expose the port publicly, start vLLM with --api-key so the endpoint is not open to anyone who finds it.

6. Benchmark before you commit. Send a few warmup requests, then measure time to first token, time between tokens, and aggregate throughput at the concurrency you actually expect. If the KV cache fills up (vLLM logs cache usage and will queue or preempt requests), the usual levers are a lower --max-model-len, a quantized checkpoint, enabling prefix caching for shared prompts, or more GPUs.

Frequently asked questions

What is vLLM used for?

vLLM is an inference engine for serving large language models. It loads a model from Hugging Face onto one or more GPUs and exposes it behind an OpenAI-compatible API, using PagedAttention and continuous batching to serve many concurrent requests from the same GPU memory.

What is the difference between prefill and decode?

Prefill processes the whole prompt in one parallel pass and writes its keys and values into the KV cache; it mostly determines time to first token. Decode then generates one token at a time, reading from that cache; it is usually limited by memory bandwidth and determines the speed of the rest of the response.

Why does the KV cache use so much memory?

Every token in the context stores a key and a value vector in every attention layer, for every concurrent request. Memory therefore grows with context length times the number of simultaneous requests, and at long context with many users the cache can exceed the size of the model weights.

Which attention backend should I use with vLLM?

Let vLLM pick automatically unless you have a reason not to. If you override it, match the backend to the model's positional encoding: FlashAttention suits RoPE models like LLaMA and Qwen, while ALiBi models like BLOOM need Torch SDPA or Triton attention.

How do I run a model that does not fit on one GPU?

Use tensor parallelism: rent a multi-GPU node and set --tensor-parallel-size to the number of GPUs. For single-node deployments this is the standard approach; pipeline parallelism is mainly for spreading very large models across several nodes. Quantization is the other lever, and often lets a model fit on fewer GPUs.

---

The fastest way to make any of this click is to watch it run: rent a GPU on the Aquanode marketplace, start the vLLM container above, and watch the KV cache numbers move as you change context length and concurrency.