Token Optimization#

Field	Value
Original Author	Sami Ksara
Version	0.0.1
Last Modified	29.05.2026
Status	OK

What Is a Token?#

A token is the smallest unit an AI model reads and writes. In English, one token is roughly 3/4 of a word — short words like "the" or "is" are a single token, while longer words get split into multiple tokens. For example, "hamburger" becomes 3 tokens. Punctuation, spaces, and code syntax all count as tokens too.

Every API call is billed by tokens in two directions: input tokens (the prompt you send) and output tokens (the response the model generates). Output tokens are typically more expensive than input tokens. Understanding this split is important because different optimization strategies target different sides of the equation.

Why Optimize?#

Cost — you pay per token. A single request might seem cheap, but agents that loop dozens of times or process large files can rack up significant costs quickly. In a team setting, this adds up fast.
Context window — every model has a maximum number of tokens it can handle in one request (prompt + response combined). Claude Sonnet supports up to 200k input tokens, for example. Once you hit the limit, content gets truncated or the request fails entirely. Long conversations and large tool outputs are the most common causes.
Speed — the more tokens the model has to process, the longer it takes to respond. Trimming unnecessary input leads to noticeably faster responses, especially in agent loops where the model is called repeatedly.

Key Strategies#

Prompt Trimming#

Remove unnecessary wording, filler, and redundant instructions from your prompts. Every token in the prompt is sent with every request, so verbosity has a compounding cost. Compare:

Before: "I would like you to please analyze the following code and provide a detailed summary of what it does and any issues you find."
After: "Analyze this code. Summarize its purpose and list any issues."

Both get the same result, but the second version uses roughly half the tokens. This applies to system prompts too — since the system prompt is included in every single API call, even small reductions there save tokens across hundreds or thousands of requests.

Conversation Compaction#

In a multi-turn conversation, the full message history is sent to the model on every new request. After 20-30 exchanges, this history can consume most of the context window, leaving little room for the actual task.

Compaction solves this by summarizing older messages into a shorter form. For example, 15 earlier turns might be condensed into a single paragraph that captures the key decisions and context. The model still has the information it needs, but the token cost drops significantly.

Most harnesses (like Claude Code) handle compaction automatically when the conversation approaches the context limit. When building your own system, you need to implement this yourself — typically by asking the model to summarize the history once it crosses a token threshold.

Output Control#

Without constraints, models tend to be verbose — they'll write five paragraphs when one sentence would do. There are two ways to control this:

Instruction-based — tell the model to be concise in your prompt. Phrases like "respond in under 50 words" or "list only the file names" work well.
Parameter-based — set the max_tokens parameter on the API call to hard-cap the response length. This is a safety net, not a formatting tool — the response will cut off mid-sentence if it hits the limit.

Controlling output length saves output tokens directly and also reduces input tokens on the next turn (since the model's previous response becomes part of the conversation history).

Tool Output Trimming#

When agents use tools — reading files, running commands, querying APIs — the raw output is fed back into the model as context. A single cat of a large log file can inject thousands of tokens into the conversation in one shot.

The fix is to trim tool outputs before they reach the model. Only pass the relevant parts: the specific error message instead of the full stack trace, the matching lines instead of the entire file, the summary instead of the raw JSON blob. This is handled at the harness level — the code that executes tools decides what to return to the model.

This is one of the highest-impact optimizations in agent systems because tool outputs are often the single largest source of token consumption.

Prompt Caching#

Many providers support prompt caching. If the beginning of your prompt is identical across multiple requests, the provider caches those tokens and charges a reduced rate for subsequent uses.

This is especially effective for:

System prompts that stay the same across every request.
Large static context blocks like documentation, code references, or knowledge bases injected into the prompt.
Multi-turn conversations where the earlier messages don't change between turns.

With Anthropic's implementation, cached input tokens cost significantly less than uncached ones. The cache has a time-to-live (TTL) — currently 5 minutes — so if you make requests within that window, you get the discount. The key requirement is that the cached content must be a prefix (the beginning of the prompt), not content in the middle or end.

When designing your prompts, put stable content first (system prompt, static context) and variable content last (user message, current task) to maximize cache hits.

Model Selection#

Not every task needs the most powerful model. A simple classification, data extraction, or formatting task can be handled by a smaller, cheaper model just as well as the flagship one.

Task complexity	Suggested model tier	Example
Simple extraction, formatting	Small (e.g. Haiku)	"Extract the email address from this text"
Summarization, general Q&A	Medium (e.g. Sonnet)	"Summarize this meeting transcript"
Complex reasoning, coding, multi-step	Large (e.g. Opus)	"Debug this failing test suite and fix the root cause"

Using the right model for each task can cut costs dramatically without sacrificing quality. In agent systems, you can even route different steps to different models — use a cheap model for simple tool-output parsing and a powerful model for the planning and reasoning steps.

Quick Reference#

Strategy	What It Saves	Impact
Prompt trimming	Input tokens every call	Medium
Conversation compaction	Input tokens as history grows	High
Output control	Output tokens per call	Medium
Tool output trimming	Input tokens per tool call	High
Prompt caching	Cost on repeated prompt prefixes	High
Model selection	Cost per token	High