Tokens & Tokenization

Why This Matters

Every interaction with an LLM is measured, billed, and limited in tokens. Your prompt is tokens in, the response is tokens out, and the context window is a fixed token budget. If you understand tokens, API bills stop being surprising, "context window exceeded" errors become predictable, and a whole class of odd model behaviors (bad arithmetic on long numbers, misspelled rare words, weird handling of URLs) suddenly makes sense.

Everyday Analogy

Imagine reading a book through a paper-towel tube that shows only a few letters at a time. You would not see whole words; you would see frequent letter chunks: "the", "ing", "tion". After enough books, you would learn which chunks follow which. That is exactly how an LLM reads: not letters, not words, but statistically common chunks called tokens.

What Is a Token?

A token is the smallest unit of text an LLM processes. Tokenizers split text into pieces drawn from a fixed vocabulary (typically 50,000–200,000 entries) learned from training data using algorithms like Byte-Pair Encoding (BPE).

The practical rules of thumb for English:

• 1 token ≈ 4 characters
• 1 token ≈ 0.75 words
• 100 tokens ≈ 75 words ≈ one short paragraph
• 1,000 tokens ≈ 750 words ≈ 1.5 pages

Common words are single tokens ("the", "hello"). Rare or long words split into several: "tokenization" becomes token + ization; a made-up word like "flibbertigibbet" might need five or six tokens.

Worked Example: Counting a Real Sentence

Take the sentence:

The quick brown fox jumps over the lazy dog.

A GPT-4-class tokenizer splits it into 10 tokens:

[The] [ quick] [ brown] [ fox] [ jumps] [ over] [ the] [ lazy] [ dog] [.]

Notice three things:

1. Spaces attach to the following word. " quick" (with its leading space) is one token, not two.
2. The period is its own token. Punctuation usually is.
3. 9 words became 10 tokens, close to the 0.75 words-per-token rule.

Now compare code:

def add(a, b):
    return a + b

This tiny function costs about 15 tokens: brackets, colons, and indentation each consume tokens. Code is consistently more token-dense than prose, which is why code-heavy prompts hit limits faster.

Tokens Drive Cost

API pricing is per million tokens, with output typically costing 3–5× more than input. A concrete budget for a customer-support bot:

At $3 per million input tokens and $15 per million output tokens, this conversation turn costs about $0.019. At 100,000 conversations per month, that is $1,900, and trimming the system prompt from 800 to 400 tokens saves $120 every month with zero quality loss if the trimmed content was redundant.

Tokenization Explains Strange Model Behavior

Several famous LLM weaknesses are tokenization artifacts, not intelligence failures:

• "How many r's in strawberry?" The model sees [str][aw][berry]: it never sees individual letters, so counting them is genuinely hard.
• Arithmetic on long numbers. 1234567 might tokenize as [123][4567]: the model does math on awkward chunks, not digits.
• Reversed words. Reversing "hello" requires letter-level manipulation the token vocabulary hides.
• Non-English costs more. Tamil, Hindi, or Japanese text can use 3–10× more tokens per sentence than English because the vocabulary is English-heavy. The same question costs more and fits less context in other languages.

Real-World Showcase

• GitHub Copilot aggressively trims context to fit token budgets, selecting only the most relevant open-file snippets because sending your whole project would cost dollars per completion.
• Claude, ChatGPT and Gemini pricing pages all quote per-million-token rates: procurement teams now forecast "token spend" the way they forecast cloud compute.
• Translation platforms route high-volume non-English workloads to models with multilingual tokenizers specifically to cut token counts, a tokenizer choice worth six figures a year at scale.

Try It Yourself

This site has a built-in tool for exactly this lesson:

1. Open the Token Counter from the Tools page.
2. Paste one paragraph of plain English. Check tokens-per-word (expect ≈ 1.3).
3. Paste the same paragraph in another language you know. Compare the count.
4. Paste a code snippet. Notice how much denser it tokenizes.
5. Type a single long word like "antidisestablishmentarianism" and watch it fragment.

Then estimate: if your daily prompt template is 1,200 tokens and you call the API 5,000 times a day at $3 per million input tokens, what is your monthly input bill? (Answer: 1,200 × 5,000 × 30 × $3 ÷ 1,000,000 = $540.)

Common Mistakes

• Estimating budgets in words or characters instead of tokens, then hitting context limits in production.
• Assuming all languages cost the same: non-English text can multiply your bill.
• Ignoring output tokens, which cost 3–5× more than input and dominate bills for long-form generation.
• Blaming the model for letter-counting failures that are really tokenization artifacts.

Key Takeaways

• Tokens, not words, are the currency of LLMs, roughly 4 characters or 0.75 English words each.
• Every cost, limit, and latency figure in the LLM world is denominated in tokens.
• Output tokens cost several times more than input tokens.
• Many "dumb model" moments are tokenizer artifacts: letter counting, long arithmetic, rare words.
• Token budgeting (trimming system prompts, capping history, selecting context) is the cheapest optimization in AI engineering.