What is a token in an LLM, and why it matters

What a token is

A token is the unit of text a language model works with. It can be a whole word, a fragment of a word, a punctuation mark or a space. Common words usually stay whole; long, rare or made-up words get split into several fragments.

Text is split into tokens before it enters the model, and each token corresponds to a number in a fixed list called the vocabulary. From that point on the model works on numbers only.

How the text gets cut

The fragments aren't picked by hand. An algorithm (byte-pair encoding is a widely used one) scans a large amount of text, finds the character sequences that appear most often and promotes them to tokens. The result is a vocabulary ranging from a few tens of thousands to around two hundred thousand pieces, depending on the model.

How many tokens is a word

It depends on the language and the model. For English, a handy rule of thumb says one token is about four characters, or roughly three quarters of a word. Other languages, especially ones less represented in the training data, need more tokens on average for the same sentence, because vocabularies are tuned mostly for English.

There is no exact number that holds everywhere: the same text produces different counts with different models.

Why tokens matter

Three practical consequences:

• The context limit: the "context window" is measured in tokens and covers both what you write and what the model answers. Beyond that threshold, the oldest parts of the conversation are no longer taken into account.
• Cost: paid APIs charge input tokens and output tokens separately.
• Speed: the answer is produced one token at a time, so long answers take longer.

The odd mistakes tokens help explain

Counting the letters in a word, spelling it backwards or doing arithmetic on very long numbers are tasks where models fail more often than you'd expect. One cause is tokenization itself: the model doesn't see a word letter by letter but as whole pieces, so it only has indirect information about individual letters.

It isn't the only cause, but it helps explain why seemingly simple questions can trip up very capable systems.

How to use this

• If a text is very long, summarize it or split it into parts instead of pasting it all at once.
• Ask for the answer in the format you need: a short table costs less than an essay.
• For counting letters or exact calculations, use a tool (a calculator, a spreadsheet) instead of trusting the answer from memory.

More guides

• How an LLM works, explained without formulas
• How to write a good prompt: five rules with examples
• Why AI makes things up, and how to check its answers