← All explainers
Explainer

Tokens: The Chunks of Text an AI Model Reads and Writes

5 min read Reviewed
A paper strip reading 'the model reads tokens.' feeds into a small machine on a wooden desk and comes out as separate tiles, with 'model' and 'tokens' each split in two. A robotic arm places the last tile, and in front lies a row of tiles reading 'AI writes back one token at a time.'

Key takeaways

  • A token is a chunk of text that a language model reads and writes as one unit: a whole word, part of a word, a single character or a punctuation mark. A token isn't necessarily a word.
  • A tokenizer splits your text into tokens and represents each one with a numerical ID. The model writes its answer one token at a time, and the tokens it picks are translated into readable text.
  • The same text can come out as a different number of tokens with a different tokenizer, and counts also depend on the language. Limits are counted in tokens, including the context window, everything a model can see while it answers.
  • On pay-per-use pricing, input and output tokens have their own prices. We recommend comparing models on token charges per usable result, on a sample of your own work, retries included. Track the time people spend checking results as its own number.

Chunks of text, in and out

A token is a chunk of text that a language model reads and writes as one unit. It can be a whole word, part of a word, a single character or a punctuation mark. Before the model works on your text, a tool called a tokenizer splits it into tokens and represents each one with a numerical ID. The model writes its answer one token at a time, and the tokens it picks are translated into readable text.

So your question goes in as tokens, and the answer comes out as tokens. You rarely see them when you chat, but the model’s limits count them, and so do pay-per-use bills.

This piece is about text and the models that write it. Images and audio can be turned into tokens too.

From a question to token IDs and back

Take the word “tokenization”. Together with the space in front of it, which belongs to the first token, one provider’s tokenizer splits it into two tokens, ” token” and “ization”, while a short, common word like ” the” stays a single token. Tokens are chunks of characters that turn up often, so common words tend to stay whole and longer or rarer ones get split.

A token is a chunk of text a model reads and writes. An example of one question going through a language model. A tokenizer splits the text What is tokenization? into five tokens: What, is, token, ization and a question mark. The tokens for is and token include the space before the word. The word tokenization becomes two tokens, highlighted: one word, two tokens. Each token is shown with an illustrative ID number. The model writes its answer one token at a time, in six steps: It, splits, text, into, tokens and a full stop, with the new token highlighted at each step. The output tokens for splits, text, into and tokens include the space before the word. A bracket over the steps loops back to the model: it uses the question and the answer so far to pick the next token. Below the separate output-token tiles, the answer is shown as readable text: It splits text into tokens. The split and the IDs are an illustration. Real splits depend on the tokenizer. A token is a chunk of text a model reads and writes. One question in, one answer out, as an example. Your text What is tokenization? Tokenizer Tokens What is token ization ? One word, two tokens. Token IDs (illustrative) 41 9 507 88 3 Each token is represented by an ID number. The model The answer, one token at a time 1 It 2 It splits 3 It splits text 4 It splits text into 5 It splits text into tokens 6 It splits text into tokens . Blue marks the new token at each step. The model uses your question and the answer so far to pick the next token. The answer as readable text It splits text into tokens. Example split and IDs. Real splits depend on the tokenizer.
Your question goes in as tokens, and the answer comes out as tokens, written one at a time and translated into readable text.

Each token’s ID identifies it in the model’s vocabulary, the set of tokens the model knows, so within that vocabulary the same token always has the same ID. The model works with those IDs. When it answers, it picks one token, adds it to the sequence so far and picks the next, choosing each one using patterns it learned in training, applied to your input and the answer so far.

Not every model splits text the same way. Different models can use different tokenizers, so the same text can come out as a different number of tokens, and counts also depend on the language the text is written in. Think of the same pizza cut with two different cutters: one gives ten pieces, the other thirteen. Same pizza, a different number of pieces. Same text, a different number of tokens.

Not words, and not a fixed size

A token isn’t necessarily a word. One word can be one token or several, so counting words can’t tell you how many tokens a text will take. As a rough guide for English, two providers put a token at about four characters, and one puts 100 tokens at roughly 60 to 80 words.

That matters because limits are counted in tokens. A model’s context window, everything it can see while it answers, has a fixed size in tokens, and what you send and what it writes have to fit inside it together. So the same limit holds less text when a tokenizer splits that text into more tokens.

Some models also work through a problem before they answer, and that reasoning is written in tokens too. Two providers charge for reasoning tokens at their output-token rates. One of them adds that you pay for them even when the reasoning isn’t shown to you, and that they take room in the context window.

The pizza only explains the split. A tokenizer cuts along the chunks in its vocabulary, not evenly, and a model’s answer isn’t cut from a finished whole: it’s written token by token.

A monthly job on two models

Say your team is choosing between two models for a job it runs every month: one summary for each of about 1,000 customer complaints. Model A charges $3 per million input tokens, the text you send in, and Model B charges $2.50. (The prices, counts and results in this example are made up.) B looks cheaper.

But the price page quotes a price per piece of input, and the number of pieces depends on the cutter. Suppose a month of complaints comes to 1 million input tokens on A and 1.3 million on B. The input then costs $3 on A and $3.25 on B: a lower price per token, a higher input cost.

Input is only part of it. On two providers’ current price lists, writing a token costs about four to eight times as much as reading one on their text models, so how much each model writes matters too, and so do retries when a summary isn’t usable.

So the team tests both on 50 real complaints covering the kinds and lengths they actually get, with the same instructions and a usable summary defined up front. On each provider’s pay-per-use access for developers, every response reports the tokens it used, so they log those and every redo. Suppose A’s 50 usable summaries come to $2 in token charges, retries included, and B’s to $2.25. That’s 4 cents against 4.5 cents per usable summary, or about $40 against $45 a month, before anyone’s time checking them. In this made-up test, B costs more per usable summary despite its lower input price. With other numbers, it could have won.

Pricing a model by the result

What you pay depends on how many tokens your work becomes and which prices apply, and some price lists charge more per token once a request passes a set length. The count moves with the tokenizer, the language, how much the model writes and reasons, earlier messages a chat tool may send again with each new one, and retries.

Our recommendation, a method and not an industry standard: compare models on token charges per usable result, on representative work, under the same quality standard, retries included, and track the time people spend checking results as its own number. A sample gives a first estimate, and a pilot on real work checks it. It’s the same logic as testing a cheaper model before booking a saving, applied when you choose.

Not every AI bill shows tokens, though. Some plans charge a fixed fee per seat for everyday use, and others bill every token on top of the seat. And a token count measures how much text was processed, not what the work achieved, which is why ranking employees by the tokens they use rewards the wrong thing.

Checks for your next model choice

  1. Which of our AI charges are seat fees, which are billed by usage, and where do we pay both?
  2. Which input and output prices apply to our work?
  3. When we compared or switched models, did we measure token counts on our own text with each one?
  4. What does one usable result cost us on a sample of our own work, retries included, and who decided what “usable” means?

Questions this explainer gets

Are tokens the same as words?

No. A token can be a whole word, part of a word, a single character or a punctuation mark. Two providers put a token at about four characters of English, and one puts 100 tokens at roughly 60 to 80 English words, but the count depends on the tokenizer and the language.

Why did our AI bill rise when the advertised token prices stayed the same?

The counts can grow even when advertised prices stay the same: a model whose tokenizer produces more tokens for the same text, earlier messages sent again, longer answers or more reasoning. Separately, some price lists charge a higher rate per token once a request passes a set length, so the rate that applies can change even when the listed prices don't.