Scaling AI Observability: How Clay Monitors Millions of…

Learn how Clay uses LangSmith to move from manual agent review to automated, large-scale monitoring using online evaluators.

Monitoring ten AI agent runs is a breeze. Monitoring ten million? That is where the real nightmare begins.

⚡ TL;DR: Clay uses LangSmith's insights and online evaluators to automate the monitoring of millions of agent runs, replacing manual review with scalable, automated observability.

The Simple Version

When you first build an AI agent, you can read every single log to make sure it's working. You are basically acting as the quality control manager for every single task.

But as your app grows, you can't possibly read millions of logs. You need a way to spot errors automatically without reading every line of text.

🎯 Think of it this way: It is like managing a lemonade stand. When you have one employee, you can watch them pour every cup. When you have a million stands worldwide, you need a digital dashboard that alerts you only when a stand is running out of lemons.

How It Actually Works

The Wall of Scale

At a small scale, manual review is the gold standard. You see exactly where the agent hallucinated or failed.

However, manual review doesn't scale linearly. Once you hit millions of runs, the volume of data becomes a "wall" that hides the actual problems.

Online Evaluators

To break through that wall, Clay uses online evaluators. These are automated scripts or other LLMs that "grade" the agent's performance in real-time.

🎯 Think of it this way: Instead of a boss watching every employee, you give every employee a rubric and a robotic supervisor who checks their work against that rubric instantly.

LangSmith Insights

By using LangSmith, Clay can aggregate these evaluations into high-level insights. This allows them to see patterns of failure across millions of runs rather than hunting for a needle in a haystack.

Technical Deep Dive

For the developers, this is a shift from qualitative analysis to quantitative observability. Instead of reading a trace, you are monitoring a metric.

Clay implements LLM-as-a-judge patterns. This involves using a more powerful model (like GPT-4o) to evaluate the output of a smaller, faster agent based on specific criteria like accuracy or tone.

These evaluators generate a score or a boolean (Pass/Fail) for every run. These data points are then streamed into LangSmith, allowing the team to run statistical queries to find exactly which prompt versions or tool-calls are causing the most regressions.

Why This Matters

AI agents are non-deterministic, meaning they can fail in unpredictable ways. Without automated monitoring, these "silent failures" can ruin your user experience before you even notice them.

Scaling observability ensures that as you add more users, your reliability increases instead of decreases. It turns the "black box" of LLM behavior into a manageable engineering problem.

Key Takeaways

Manual review fails at scale: You cannot hand-check millions of agent runs.
Observability is mandatory: Production-grade agents require a system to track performance.
Online Evaluators: Use automated graders to score agent outputs in real-time.
LLM-as-a-Judge: Leverage powerful models to audit the performance of smaller agentic workflows.
Pattern Recognition: Tools like LangSmith turn individual traces into aggregate insights.
Prevent Silent Failures: Automated monitoring catches regressions that humans would miss.

Scaling AI Observability: How Clay Monitors Millions of Agent Runs