---
title: "Token counts are not word counts"
url: https://aimodelscompare.com/notes/token-counts-are-not-word-counts/
description: "Estimating usage from character or word length is convenient and reliably wrong at the margins."
---

# Token counts are not word counts

17 AUG 2026 · 08:05 · CONTEXT

Estimating usage from character or word length is convenient and reliably wrong at the margins.

Billing, context limits and truncation all operate on tokens, and tokens are produced by a learned segmentation of text, not by splitting on spaces. Ordinary English prose tends to land near a stable ratio of tokens to words, which is exactly what makes the shortcut tempting, and exactly why it fails silently when the input stops being ordinary English prose.

## Where the ratio breaks

Segmentation cost rises for text the tokenizer was not shaped around: identifiers and code, chemical or legal notation, long digit strings, URLs, base64 blobs, emoji and most non-Latin scripts. A line of source code can consume several times the tokens its character count suggests, because punctuation, indentation and camel-cased names each fragment into multiple pieces. The same document translated into a script with poorer coverage can cost a multiple of the original without a word of content changing.

## What to do instead

Count tokens with the tokenizer that belongs to the candidate you are testing, on your own sample of real inputs rather than on a paragraph of clean prose. Do it once, early, and record the distribution rather than the mean: the tail is what overruns a context limit, and the tail is exactly what a mean hides. Where two candidates use different tokenizers, note that a comparison of their per-token prices is not a comparison of their per-request prices.

## Terms used in this note

[token](https://aimodelscompare.com/glossary/) · [tokenizer](https://aimodelscompare.com/glossary/) · [context window](https://aimodelscompare.com/glossary/) · [truncation](https://aimodelscompare.com/glossary/)

[Back to the river](https://aimodelscompare.com/) · [More in Context](https://aimodelscompare.com/topics/context/)