What is Tokenization in AI? How LLMs Process Text, Calculate Costs & Optimization

1. How Tokenization Works: The Step-by-Step Pipeline

Tokenization converts raw text strings into numerical tensors through a 4-stage processing pipeline:

2. Byte-Pair Encoding (BPE): The Standard Tokenization Algorithm

Why don't LLMs just use whole words or individual characters?

• Word-level tokenization: Requires an impossibly massive vocabulary (millions of entries) to cover every language, slang, and typo, and cannot handle out-of-vocabulary words.
• Character-level tokenization: Keeps vocabulary small (around 256 entries), but results in massive token sequences, causing models to run out of context window and computing memory extremely fast.

Byte-Pair Encoding (BPE) strikes the perfect balance. It starts with individual characters and iteratively merges the most frequently occurring character pairs in a training dataset until it reaches a target vocabulary size (typically 32,000 to 100,000 tokens).

3. Token Pricing & Practical Optimization Strategies

Every API request to models like GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro incurs costs based on token count. Here is how to keep token spend under control:

The Model Behaviours Tokenisation Explains

Tokenisation is worth understanding not as a piece of technical trivia but because it cleanly accounts for a specific and recurring set of model failures that otherwise look like gaps in intelligence.

Counting letters. Asking a model how many times a particular letter appears in a given word is famously and consistently unreliable, and the reason is representational rather than cognitive. The model received an identifier for a chunk of text, not a sequence of characters. Asking it to count letters is roughly like asking someone to count the strokes in a word they only ever heard spoken.

Reversing strings and working with spelling. Same cause. Any task requiring manipulation at the character level is being performed on a representation that does not expose characters. Models often succeed anyway by having memorised spellings from training text, which makes the failures inconsistent and therefore more confusing.

Arithmetic on long numbers. Numbers tokenise inconsistently — a figure may split into fragments that do not align with digit positions, so the model is not seeing place value the way a person reading it would. This is part of why calculation reliability degrades with digit count and why routing arithmetic to a tool is standard practice rather than an admission of defeat.

Sensitivity to formatting. An extra space, an unusual line break or trailing whitespace changes the token sequence, and occasionally changes behaviour in ways that seem disproportionate to the edit. This is not superstition about prompt formatting; it is a real consequence of the input being a different sequence of identifiers.

Recognising this category saves a great deal of wasted effort. These failures do not improve with better prompting, because the information the task requires never reached the model. The correct response is to handle the operation outside the model — in code, with a tool call — rather than trying to coax the model into doing something its representation forecloses.

Why Models Use Tokens Rather Than Words or Characters

Tokenisation looks like an arbitrary implementation detail until you consider the alternatives, at which point it becomes clear why every production language model uses subword tokens rather than something more intuitive.

Words are the obvious choice and fail badly. A vocabulary of whole words has to be enormous to cover a language, and it still cannot handle anything outside it — new terms, product names, typos, technical jargon, or any word in a language you did not anticipate. Every unseen word becomes an unknown token, which discards information entirely. Morphologically rich languages make this dramatically worse, since a single root generates dozens of inflected forms that would each need their own entry.

Characters are the other extreme and fail differently. A character vocabulary is tiny and can represent anything, but sequences become very long — a paragraph that is a few hundred tokens becomes a few thousand characters. Since attention cost grows sharply with sequence length, character-level modelling is computationally expensive, and the model must also learn word structure from scratch rather than being given it.

Subword tokens are the compromise that works. Common words become single tokens; rare words decompose into meaningful fragments. Nothing is ever unrepresentable, because the fallback decomposes toward individual characters. Sequence lengths stay manageable. The vocabulary stays a fixed, tractable size. This is why the approach became universal rather than being one option among several.

The consequence worth internalising is that the model never sees letters. It sees identifiers for chunks of text, and any task requiring character-level awareness — counting letters, reversing a word, detecting a rhyme, working with precise string manipulation — asks the model to reason about something it has no direct access to. Those famous failure cases are not gaps in reasoning; they are gaps in representation.

How a Tokeniser Learns Its Vocabulary

The vocabulary a model uses is not designed by hand by a linguist. It is learned from a corpus by an algorithm, and understanding roughly how that works explains most of the behaviour people find surprising.

The dominant approach, byte pair encoding, begins with individual characters and repeatedly merges the most frequently co-occurring pair into a new single token. Starting from letters, it might notice that t and h appear together constantly and merge them into th. Then th and e become the. The process repeats for tens of thousands of iterations until the vocabulary reaches its target size.

Three consequences follow directly from that mechanism. Frequency determines efficiency — text resembling the training corpus compresses well, and text unlike it does not, purely because the merges were optimised for what the algorithm saw. Common words become single tokens while rare ones fragment, which is why an everyday word costs one token and a technical term or unusual name may cost five. And nothing is ever unrepresentable, because if no merge applies the text decomposes to individual bytes.

A detail that catches people out: leading spaces are part of tokens. In most tokenisers, the word appearing after a space and the same word at the start of a line are different tokens with different identifiers. This is why prompt formatting can subtly change behaviour, and why trailing whitespace in a prompt occasionally produces oddly degraded output — the model is being asked to continue from a token boundary that rarely occurs in natural text.

Casing behaves similarly. A capitalised word and its lowercase form are usually distinct tokens, which is one reason models can be sensitive to formatting in ways that feel arbitrary. None of this is a defect; it is the direct result of learning a vocabulary statistically rather than defining one linguistically.

Tokenisation Is Not Equally Efficient for Everyone

Tokenisers are trained on a corpus of text, and the composition of that corpus determines which languages and formats compress efficiently and which do not. The result is a real and under-discussed inequality in what different users pay for the same information.

English text typically compresses to roughly three-quarters of a token per word, because English dominates most training corpora and common English words earned their own entries. Languages that appeared less often in that corpus — and particularly those using non-Latin scripts — fragment into far more tokens for the same meaning, sometimes by a multiple rather than a margin.

This has three practical consequences that compound. Cost: since billing is per token, conveying the same information in some languages costs several times more. Context: the effective context window shrinks proportionally, so a document that fits comfortably in English may not fit in another language. Latency: more tokens means more generation steps and slower responses.

Code and structured data have their own characteristics. Whitespace-heavy formatting, deeply indented JSON and repetitive markup consume tokens without conveying much information, which is why minifying data before sending it to a model is a genuine cost optimisation rather than a micro-optimisation. Long identifiers, UUIDs and hashes fragment badly because they are effectively random strings the tokeniser has never seen.

For anyone building products serving multiple markets, this is worth measuring rather than assuming. Run representative text from each language through a tokeniser and compare token counts for equivalent content. The results frequently change assumptions about which markets are economically viable at a given price point, and that is a business finding rather than a technical one.

Tokens, Context Windows and What Actually Fits

Context window sizes are always quoted in tokens, and translating that figure into something practically meaningful requires a couple of conversions that people rarely bother making until something fails unexpectedly in production.

A useful rough conversion for English is that a token averages around three-quarters of a word, so a hundred thousand tokens is roughly seventy-five thousand words — a decent-sized book. That sounds generous until you account for what else occupies the window. The system prompt, the conversation history, any retrieved documents, tool definitions and the model's own response all draw from the same budget.

Two constraints catch teams out. The first is that output shares the window in most implementations — filling the context with input leaves insufficient room for a substantial answer, which manifests as truncated responses rather than an error. The second is that the effective window is smaller than the stated one. Models attend less reliably to material in the middle of a long context, a well-documented effect meaning that information buried in the centre of a very long input is frequently treated as though it were absent.

That second point has a direct practical consequence: filling a large window is not the same as using it. Instructions belong near the beginning or the end, the most relevant retrieved material should be ranked toward the edges rather than buried, and a smaller, better-selected context frequently outperforms a larger one. This is the underlying reason reranking in retrieval systems improves accuracy while also reducing cost — fewer, more relevant tokens beat more, less relevant ones on both dimensions simultaneously.

It is also why very large context windows did not eliminate retrieval. The ability to paste an entire corpus into a prompt exists; the ability to have the model attend to all of it equally does not, and the economics of doing so on every request remain unattractive.

Managing What Tokens Cost You

Tokens serve as the unit of billing, the unit of latency and the unit of context capacity all at once, which means token discipline is one of the few optimisations that improves all three at once.

The largest recurring cost in most production systems is the system prompt, because it is sent on every single request. A two-thousand-token instruction block on a million monthly calls is two billion tokens a month spent on text that never changes. Prompt caching, where supported, addresses much of this and is usually the single highest-return change available. Shortening the stable instructions helps regardless.

Retrieved context is the second major line, and it is where reranking pays for itself twice. Passing twenty retrieved passages when three would do is both more expensive and less accurate, since irrelevant context dilutes attention. A reranker that reduces what you send frequently improves quality while cutting cost, which is an unusually favourable trade.

Conversation history is the one that grows without anyone noticing. Each turn re-sends everything before it, so a long conversation costs quadratically rather than linearly. Systems that keep full history indefinitely become expensive in ways that only appear on the invoice. Summarising older turns, or dropping ones that are no longer relevant, is standard practice for anything long-running.

Output tokens generally cost more than input tokens, and generation is where latency lives. Asking for a concise answer is not only a quality choice but a cost and speed one, and instructing a model to omit preamble and restatement is worth doing explicitly in high-volume systems.

Structured output formats deserve a mention because the choice is quietly expensive at volume. JSON is verbose — every key is repeated on every object, and braces, quotes and commas all consume tokens without carrying information. For a response returning many similar records, a more compact representation can cut output tokens substantially, and the model handles it perfectly well provided the format is specified clearly. This is not premature optimisation when the same call runs a million times a month.

A final practical note: estimate rather than guess. Token counting libraries let you measure a representative request before deployment, and the difference between an estimated and an actual monthly bill is frequently an order of magnitude when nobody checked. For deeper context on where these costs sit in a wider system, our guide to prompting covers the trade-offs between prompt length and reliability.