Home › Glossary › Token
Core · beginner

What is Token?

A plain-English explanation of Token (Token) — what it means, why it matters, and how it is used in AI.

Token
Token
A token is the basic unit of text that a language model processes. Tokens are roughly equivalent to word pieces — common words are single tokens, while longer or rarer words may be split into multiple tokens.
"The sentence "ChatGPT is amazing" contains approximately 4 tokens. A typical page of text contains around 500 tokens."

Also known as: Token, subword token, wordpiece

Context WindowLLMPrompt

Why does Token matter?

Token limits determine how much text you can send to a model at once. Token counts determine API costs.

Practice this term

The best way to remember Token is to practice unscrambling it. AI Terminology Scrambler uses spaced repetition to help you learn and retain AI vocabulary in just a few minutes a day.

Practice Token now →

Related AI terms

Context WindowLLMPrompt