We are building AI agents that can navigate our computers and solve complex tasks, but we have a massive problem: we have no idea if they're actually worth the cost.
The Simple Version
Imagine you're trying to sell a steam engine to someone who has only ever used horses. You can't talk about thermodynamics or PSI; they won't understand it.
James Watt solved this by inventing 'horsepower.' It wasn't a perfectly scientific measurement, but it was legible—it told the customer exactly how many horses the machine replaced.
AI agents have the same problem. We talk about 'token counts' and 'context windows,' but business owners don't care about tokens. They care about 'horsepower'—like how many support tickets were actually closed.
How It Actually Works
The Token Trap
Most companies are stuck in a cycle of overspending and underusing. They burn through thousands of dollars in tokens, realize they aren't seeing a clear ROI, and then slash their budget in a panic.
The mistake is treating tokens as the goal. Tokens are just an output, like the amount of gasoline a car burns. The real value is the outcome, like the destination you reached.
The Review Bottleneck
AI has made production nearly instant, but verification is still human-speed. We can generate a 500-line code diff in seconds, but a senior engineer still needs 20 minutes to review it.
This creates a paradox: the faster the agent works, the more the human bottleneck grows. Value isn't judged by how fast the code is written, but by how fast it can be safely merged.
Technical Deep Dive
Computer Use Models
Most agents rely on APIs or the Model Context Protocol (MCP). However, computer use models (like those built by Yutori) operate differently. They drive the computer exactly like a human would—clicking buttons and typing in fields—meaning they can work on any software, even if no API exists.
The Entropy Matrix
To decide if a task is suitable for an agent, Maximillian Piras suggests using an Entropy Matrix based on two axes of uncertainty:
- Step Uncertainty: How unpredictable are the steps to complete the task?
- Criteria Uncertainty: How hard is it to define what a 'correct' result looks like?
If step uncertainty is too low, just write a script. If criteria uncertainty is too high, the cost of verifying the agent's work is as high as doing the work yourself.
The 'sweet spot' is a task that resembles an NP problem: it is computationally expensive to solve, but computationally cheap to verify. When you find these tasks, you can even point a second agent at the verification process to remove the human bottleneck.
Why This Matters
Without a way to measure 'Mousepower,' AI agents will remain expensive toys for early adopters rather than core business infrastructure.
By shifting the focus from token-maxing to outcome-tracing, companies can stop the cycle of austerity and actually scale their agentic workflows.
The goal isn't to make the agent smarter, but to make the value it provides legible to the people paying the bill.
Key Takeaways
- Tokens are outputs, not outcomes: Stop measuring cost per token and start measuring cost per resolved ticket or bug.
- Legibility > Accuracy: Like 'horsepower,' we need simple metrics that non-technical stakeholders can understand.
- The Review Bottleneck: The constraint has shifted from creating content to verifying it.
- The Entropy Matrix: Only use agents for tasks with high step uncertainty but low criteria uncertainty.
- Target NP-like Tasks: Focus on workflows that are hard to execute but easy to check.
- Computer Use: UI-driven agents bridge the gap where APIs don't exist.