Anthropic's Strategy for Giving LLM Tokens Specific Jobs

The Simple Version

In the world of AI agents, we often assume that every token is the same. We think that if we just give a model a larger window or more iterations, it will eventually stumble upon the right answer. This is the belief that tokens are fungible, meaning one token spent on 'thinking' is the same as one token spent on 'doing.'

Anthropic argues this is a mistake. Instead of just increasing the budget, they suggest assigning specific roles to your tokens. It is the difference between hiring ten people to all try to solve a puzzle at once versus hiring one person to solve it, one to give advice, and one to check the work.

How It Actually Works

To move beyond simple execution, Anthropic introduces a framework where tokens are partitioned into specialized roles. This prevents the agent from getting stuck in a 'hallucination loop' where it simply repeats the same mistake with more words.

The Advising Strategy

In an advising setup, the system splits the agent into two distinct roles: the Executor and the Adviser. The executor handles the bulk of the task, but it has the permission to 'call out' to the adviser mid-task when it hits a roadblock.

The adviser doesn't do the work; it provides strategic guidance. This separation of concerns ensures that the agent can step back and evaluate its own path without losing the context of the primary goal.

The Grading Strategy

Grading is about defining 'what good looks like' before the work even starts. The system creates a detailed rubric—a set of strict requirements the final answer must meet.

A dedicated 'Grader' agent then scores each attempt against this rubric. If the score is too low, the grader sends the executor back to iterate, providing specific feedback on where the failure occurred.

The Dreaming Strategy

Dreaming is a meta-cognitive approach to agentic workflows. Instead of just trying again, the agent reads through its own previous transcripts to find patterns of failure.

It then writes these findings into a persistent memory. When the next run starts, the agent doesn't start from zero; it starts with a set of 'lessons learned' from its own past mistakes.

Technical Deep Dive

The effectiveness of these strategies was tested on a financial analysis benchmark. The team held the budget constant at roughly 600,000 tokens to ensure a fair comparison between strategies.

The results were stark. The baseline agent, which did nothing but execute, scored a 76. However, the agent that utilized the advising strategy jumped to a score of 89. This proves that the allocation of tokens is more impactful than the volume of tokens.

The Fallacy of '80% Accuracy'

Katelyn Lesse and Angela Jiang highlight a critical point regarding high-stakes data, like Profit and Loss (P&L) statements. In finance, an answer that is 80% accurate is not 'mostly useful'—it is completely useless.

If a cost number is wrong in a financial report, the entire document is compromised. You cannot make a business decision based on a report that is 'mostly' correct. Therefore, the only metric that matters is a perfect run.

Calculating the True Cost

When the benchmark was rescored based on perfect runs, the execute-only baseline passed only 42% of the time. This means that to get one clean, reliable answer, you would need to run the agent roughly three times.

Mathematically, this spikes the 'true cost' from 600,000 tokens to approximately 1.8 million tokens per successful answer. By using advising and grading strategies, Anthropic found they could reach that same level of reliability for significantly fewer tokens.

Why This Matters

This shift in thinking changes how we build AI agents for professional industries. For years, the industry trend has been to increase context windows (like Gemini's 2M tokens) or use massive prompts. Anthropic is suggesting that architectural roles are more important than raw capacity.

For developers, this means the 'tuning' process should move away from prompt engineering and toward workflow engineering. Instead of asking 'How can I make the prompt better?', you should ask 'Which job does this token need to perform?'

Ultimately, this approach reduces the cost of reliability. By investing tokens in grading and advising, we can move AI from a 'probabilistic toy' that is usually right to a 'deterministic tool' that is reliably perfect.