Every large language model reads and writes in tokens, not words and not characters. It's the unit everything else is built on: context window limits, API pricing, rate limits, even how well a model handles a language it wasn't trained heavily on. Understanding what a token actually is clears up a surprising amount of confusion about how LLMs behave.
TL;DR: A token is the smallest chunk of text a model processes as one unit, roughly 4 characters or three-quarters of a word in English. Tokenization turns text into a sequence of these chunks and maps each one to a numeric ID the model can compute on. Different languages, token types (text, punctuation, special control tokens), and context-window limits all follow from this one mechanic.
What is a token in AI
What is a token?
A token is the smallest unit of text an AI model actually operates on. Tokenization is the process of breaking a larger piece of text down into these units before anything else happens. Depending on how the tokenizer was built, a token can be a full word, a fragment of a word, or even a single punctuation mark.
Tokens don't always land on clean word boundaries. "Unbreakable" might come out as "un" plus "breakable," or split some other way entirely, depending on which fragments the tokenizer's vocabulary considers common enough to deserve their own slot. A few widely repeated rules of thumb, taken from OpenAI's own tokenizer documentation, are worth memorizing:
- 1 token is roughly 4 characters of English text
- 1 token is roughly three-quarters of a word
- 100 tokens is roughly 75 words
Those ratios shift once you leave English. Languages with different scripts, or with grammar that packs more information per word, commonly tokenize into more tokens per word than English does, largely because the vocabularies used by today's major tokenizers were built on training data that skews heavily English. That's a real, measurable cost: the same sentence can burn through more of your context window and cost more per API call purely because of which language it's written in, independent of how "hard" the request actually is.
This is also why token counts and character counts diverge more than people expect. A 4,000-character document isn't reliably "1,000 tokens" the moment it includes code, non-English text, unusual formatting, or a lot of numbers, since digits, symbols, and rare strings often each consume their own token instead of packing efficiently the way common English words do. Anyone budgeting a prompt against a hard context limit is better off running the actual text through the target model's tokenizer than estimating from character count alone.
What's happening underneath: tokens as the entry point to NLP
Natural language processing exists to let a model turn human text into something it can compute over, and tokenization is the very first step in that pipeline. After a sentence is broken into tokens, those tokens get converted into numeric vectors called embeddings, since every layer past this point in a neural network is arithmetic on numbers, not string manipulation.
This wasn't always done the way modern LLMs do it. Earlier NLP systems leaned on simpler statistical methods like Term Frequency-Inverse Document Frequency (TF-IDF) and Bag-of-Words, which counted word occurrences but had no real sense of meaning or word order. Word2Vec and GloVe were a step forward, learning dense vector representations where similar words end up near each other in the embedding space, but those vectors were static: the word "bank" got one fixed vector no matter whether the sentence was about a river or a checking account. Transformer-based models like BERT solved that by generating context-sensitive embeddings, where the same token's vector shifts depending on the surrounding words.
| Technique | Description | Strengths | Weaknesses |
|---|---|---|---|
| TF-IDF | Weighs words by how often they appear in a document relative to the whole corpus | Cheap, effective for document classification | No sense of meaning, context, or word order |
| Bag-of-Words | Counts word occurrences, ignoring grammar and order | Simple, fast to compute | Same blindness to meaning and context as TF-IDF |
| Word2Vec | Neural embeddings that place similar words near each other | Captures real semantic similarity | One fixed vector per word regardless of context |
| GloVe | Combines local and global co-occurrence statistics into vectors | Strong at capturing broad corpus-level relationships | Still static, one vector per word |
| BERT (transformer embeddings) | Generates a different vector for the same word depending on context | State-of-the-art context sensitivity | Needs far more compute to train and run |
Common tokenization techniques
The three approaches that show up across NLP and LLM tooling:
Word tokenization splits on whitespace and punctuation, so "language models are useful" becomes ["language", "models", "are", "useful"]. It's easy to reason about, but breaks down on compound words, contractions, and any language that doesn't put spaces between words.
Character tokenization treats every individual character as a token. It sidesteps the out-of-vocabulary problem entirely, since any string can be spelled out character by character, but it multiplies sequence length and strips away word-level meaning that the model would otherwise get for free.
Subword tokenization splits rare or complex words into smaller recurring fragments while keeping common words whole, so "unbreakable" might become ["un", "break", "able"]. This is the default in essentially every modern LLM, because it keeps vocabulary size manageable while still handling words the tokenizer never saw during training.
| Technique | Description | Weaknesses | Use cases |
|---|---|---|---|
| Word tokenization | Splits on word boundaries | Struggles with compounds, contractions, and languages without spaces | Simple classification, keyword search |
| Character tokenization | Splits into individual characters | Long sequences, weak semantic signal | Languages without word boundaries, typo-tolerant models |
| Subword tokenization | Splits into learned common fragments | More complex to train (BPE, WordPiece, SentencePiece all differ) | Nearly all modern LLMs: GPT, Llama, BERT, multilingual models |
Types of tokens used in LLMs
Not every token in a model's vocabulary represents a chunk of readable text.
Text tokens
The bulk of any vocabulary: whole words or word fragments that carry the actual content of a sentence. "AI is useful" might tokenize as ["AI", " is", " useful"], or split further depending on the tokenizer.
Punctuation tokens
Commas, periods, exclamation points and similar marks each typically get their own token. They preserve sentence structure and rhythm; without them, generated text would run on without any signal of where one thought ends and the next begins.
Special tokens
These don't correspond to any visible text at all. They exist purely to control model behavior:
- An end-of-sequence marker tells the model (or the code sampling from it) that generation is complete.
- A newline token represents a line break.
- Padding tokens fill out shorter sequences so a batch of inputs can share a uniform length during training or serving.
- Separator tokens mark boundaries between parts of a structured prompt, such as where a system instruction ends and a user message begins.
LLM token limits (context windows)
Every model caps how many tokens it can hold in its context window at once, spanning the prompt, any system instructions, and the response together. A larger context window means the model can track more of a conversation or a longer document without losing earlier information, but a bigger number on a spec sheet doesn't automatically mean better answers: a model with a smaller, well-utilized window can still outperform one with a huge window it doesn't use efficiently.
| Model family | Published context window | Typical fit |
|---|---|---|
| Llama 3 (8B/70B) | 8,192 tokens | Short-to-medium chats, single-document summarization |
| GPT-3.5 Turbo | Up to 16,385 tokens | Longer dialogues, moderate document analysis |
| GPT-4 Turbo | Up to 128,000 tokens | Long documents, large codebases, extended agent sessions |
| Claude 3 family | Up to 200,000 tokens | Book-length documents, very long multi-turn sessions |
Sources: model context-window figures are published by each vendor (Meta's Llama 3 model card, OpenAI's model documentation, Anthropic's Claude documentation); check current limits before budgeting, vendors update them.
Once you know roughly how many tokens your prompts and expected outputs will consume, it's worth pairing that against actual GPU memory if you're self-hosting rather than calling an API: our guide on sizing VRAM for LLMs walks through exactly how context length and KV cache size trade off against the GPU you rent.
Tokenization challenges
Language resists clean rules, and tokenizers inherit every one of those messy edge cases.
Ambiguity
The same string can mean different things depending on context, and tokenization alone can't resolve that. "Apple" the fruit and "Apple" the company tokenize identically; only the surrounding context, handled later in the model, disambiguates them. Compound phrases cause similar trouble: "hot dog" the food versus "hot" plus "dog" describing an overheated animal are the same three tokens in a different arrangement of meaning that tokenization has no way to see.
Language boundaries
Languages like Chinese, Japanese, and Thai don't separate words with spaces, which breaks any tokenizer that relies on whitespace as a boundary signal. A tokenizer for these languages needs statistical segmentation or a learned vocabulary to guess where one word ends and the next begins, and getting it wrong can silently corrupt downstream tasks like search or translation. Some languages create the opposite problem: German famously builds long compound nouns by concatenation, and a tokenizer has to decide whether to keep a compound intact or split it into its component parts.
Edge cases
Phone numbers, email addresses, URLs, and hyphenated words all sit outside the assumptions most tokenizers were designed around. A naive tokenizer might split a phone number into disconnected fragments, losing the structure that made it recognizable as a phone number in the first place. An email address or a URL faces the same problem: a tokenizer that has no reason to treat "user@example.com" as one coherent unit may fragment it in a way that makes exact-match tasks, like extracting contact details from text, unreliable. Acronyms and hyphenated compounds add another layer of ambiguity, since whether "U.S.A." or "self-esteem" should be one token or several depends entirely on context the tokenizer doesn't have access to at split time.
Code brings its own edge cases too: variable names, operators, and indentation all need to survive tokenization intact for a model to reason about a program correctly. A tokenizer trained mostly on prose can split snake_case_variable or a chain of operators in ways that scramble a program's structure before the model ever sees it, which is one of the reasons code-focused models typically ship with their own tokenizer rather than reusing a general-purpose one.
Getting more out of every token
Tokens are the currency AI actually runs on, and treating them as an afterthought shows up directly in your bill and your model's output quality. Keeping prompts concise and well-structured conserves tokens without losing information, and requesting a specific output format, like a table or bullet list, from a model often uses fewer tokens than an open-ended prose response containing the same facts.
As tokenization gets applied further outside chat interfaces, into coding agents, tool-using systems, and long-running pipelines, understanding how your input actually gets counted matters more, not less. If you're running inference workloads where token throughput directly determines your GPU bill, our breakdown of how AI inference actually works covers what happens to those tokens once they leave the tokenizer, and our tokenizer deep dive covers how to build one of your own.
Frequently asked questions
How do token limits affect conversational AI performance?
Token limits set a hard ceiling on how much of a conversation a model can hold in view at once. Once a conversation grows past that limit, earlier turns have to be dropped or summarized, which is why long chats with a smaller-context model can start to lose track of details from earlier in the session. Larger context windows delay that problem but don't eliminate the underlying tradeoff between context length, latency, and cost.
What are padding tokens and why do models need them?
Models process inputs in batches for efficiency, and every sequence in a batch has to share the same length for that batching to work. Padding tokens fill the gap in shorter sequences so they match the length of the longest one in the batch, without those filler tokens ever being treated as meaningful content by the model.
Why do code and natural language sometimes use different tokenization approaches?
Programming languages follow strict, unambiguous syntax rules that natural language doesn't, so a tokenizer built for prose can split variable names, operators, and indentation in ways that obscure a program's structure. Tokenizers built or fine-tuned specifically for code preserve those syntactic units instead, which is part of why code-specialized models tend to complete and debug code more reliably than general-purpose language tokenizers asked to do the same job.
