Skip to content

Plain English

What is a token?

Tokens, token limits, price per token. It sounds technical, but the idea is simple.

2 min read

01What it is

Before an AI reads your message, it cuts it into small pieces. Each piece is a token.

A token can be a whole word, part of a word, or even a single character. That's how Anthropic's glossary puts it: the text is turned into a row of tokens, and only then does the model process it.

Model
The AI program that reads your message and puts the answer together.
Tokenizer
The part that cuts the text into tokens. Each model can cut it differently.

02Where you hear it

  • Price per token: companies charge for their models per million tokens, with what goes in (your message) and what comes out (the answer) priced separately.
  • Token limit: how much text a model can keep in mind at once. It's called the context window, and it's measured in tokens.
  • Max output: how much it can write in one go, also in tokens.

03An example

The official page for Claude Fable 5.2 says it has a 1 million token window, and it spells out how much that is:

≈ 555,000 wordsfit in 1 million tokens, according to Anthropic's docs (English text, with their current tokenizer).

On that same page, the price is USD 8 per million tokens in and USD 40 per million tokens out. Answering costs more than reading.

My own comparison: it's like a taxi meter. The longer the ride, the more you pay. Here the ride is the text going back and forth.

04Why it matters

The same sentence doesn't always come out to the same number of tokens. It depends on the model and the language. Anthropic's glossary says its models from Claude 4.7 on use a new tokenizer that produces about 30% more tokens for the same text.

So if you compare prices between models, the price per million isn't enough: how many tokens each one gets out of your text counts too. To know exactly, Anthropic recommends counting them with the model you're going to use.

05Sources

The official pages where I read each fact, with the date I saw them.

  1. Anthropic: glossary (Tokens)platform.claude.com/docs/en/about-claude/glossary
  2. Anthropic: Claude Fable 5.2 overviewplatform.claude.com/docs/en/models/fable-5-2/overview

Keep reading

Newsletter

What I'm trying, explained simply, straight to your inbox. In Spanish for now. No spam, and you can unsubscribe anytime.

Subscribe