Foundations

Token

The basic unit of text that a language model processes — typically a word, subword, or character.

LLMs don't read words — they read tokens. A token is typically 3-4 characters of English text. The word 'tokenization' becomes 3 tokens: 'token', 'ization' (or similar split depending on the tokenizer).

Token counts matter because they determine model context limits, API costs, and processing speed. A model with a 200K context window can process roughly 150,000 words at once.

← All terms