token토큰
The basic unit of text an AI model processes — often smaller than a word.
LLMs read and write text as pieces called tokens rather than letters or words. In English a word is roughly one or two tokens; languages like Korean often need more tokens for the same text.
Both what AI services charge and how much they can handle at once (the context window) are measured in tokens.
For example
The word "unbelievable" might be split into tokens such as "un", "believ" and "able".
Related terms
Where this term appears
- New US lab Reflection releases Beam, a 501B-parameter open modelModels & products
- Apple's MLX runs an 816B-parameter model on four Mac StudiosResearch
- Google launches Gemini 4 Argon, a frontier model aimed at coding and professional workModels & products
- Claude finds a new enzyme system with CRISPR-like DNA repeatsResearch
- Google's EmbeddingGemma 2 brings on-device search across text, images, audio and videoModels & products
- Mistral previews Mistral Large 4, a 1-trillion-parameter open-weight modelModels & products
- What is an LLM? A five-minute explainerExplainer
- Tokens and pricing, explained simplyExplainer
- Your first LLM API call: from key to responseHow-to guide
- Get structured (JSON) output from an LLMHow-to guide
- Cut API costs with prompt cachingHow-to guide
- A minimal RAG chatbot with ten documentsHow-to guide