Tokenizer Visualizer

What this tool does

You paste text and see exactly which tokens the model splits it into, each in a different colour with its numeric identifier. It is not an estimate: it is the same tokeniser current OpenAI models use, running in your browser.

If what you need is the cost in money, that is the token counter. This is for understanding why that number is what it is.

What a token actually is

Neither a letter nor a word: a frequent fragment the tokeniser learned from its training corpus. The practical rules become obvious as soon as you start playing:

• A common English word fits in a single token.
• A rare, technical or foreign word splits into several.
• The space attaches to the start of the following token, not the end of the previous one. That is why the text view shows ·word with the interpunct in front: that space is part of the token.
• Long numbers, identifiers and URLs get chopped up mercilessly.

That last rule explains several oddities at once: a UUID that looks like a short piece of data to you can cost twenty tokens, and a JSON blob with long keys repeated hundreds of times is far more expensive than it looks.

Spanish costs more, and you can see it

This is the main reason the tool exists, and it is easy to verify: press the two example buttons and compare the counters.

Tokenisers were trained overwhelmingly on English text. The consequence is that frequent English words get their own token, while Spanish ones — with their accents, longer endings and smaller presence in the corpus — are split into pieces.

An example you will see in the tool itself: tokenización splits into token + ización. prueba is one whole token, but internacionalización is several.

Translated into billing: the same content in Spanish consumes more tokens than in English, and you pay for that twice — in the prompt you send and in the response you get back. If you are building anything with volume on a paid API, that is a real difference, not a curiosity.

There is no trick to dodge it entirely, but knowing it changes decisions: where to trim context, whether summarising before sending is worth it, and how optimistic your cost estimate is if you calculated it with English text.

Why models fail at counting letters

The classic question — “how many r’s are in strawberry?” — is answered wrong with surprising frequency, and here you can see why: the model never saw the individual letters. It saw two or three fragments. Asking it to count characters is asking it to reason about something absent from its input representation.

The same goes for reversing words, counting syllables or wordplay. It is not that the model is stupid at those: it is looking at something else.

Which tokeniser this is

o200k_base, the one behind current OpenAI models. The count is exact for them.

Anthropic and Google use their own tokenisers and do not publish an equivalent local implementation, so their figures vary a little — the token counter applies an adjustment factor for those cases and says so. But the behaviour you observe here is common to all of them: what splits and what does not, the leading space, and Spanish coming out more expensive.

What it does not do

• It calls no API. The tokeniser downloads to your browser and the computation happens there; you can confirm it in the network tab.
• It does not calculate prices. That is the token counter, which also maintains the pricing table.
• It does not count image or audio tokens. Multimodal models bill those separately and with different logic.