Tokenization

Executive Definition: Tokenization — The process of breaking text into smaller, computationally manageable units called tokens for an AI model to process.

Tokenization converts text into discrete chunks, which can be whole words, syllables, or single characters. Token counts directly impact the context window limits and the cost of querying large language models.

Frequently Asked Questions

How do tokens affect AI costs?

Most APIs charge per token for both input (prompts) and output (completions). Understanding token counts is crucial for cost optimization.

Are tokens standardized across all models?

No. Each model family uses its own tokenizer. The exact same text will yield different token counts depending on the model (e.g., GPT-4 vs. Claude).

What is a context window?

The maximum number of tokens a model can process in a single request, including both the prompt and the generated response.