Clay’s 300M‑Run Agent Stack: Eval‑First Scaling with…

Explore Clay’s 300M‑run agent pipeline, its eval philosophy, data‑lake architecture, and how LangSmith enables safe scaling.

Curious how a startup built a 300‑million‑run agent pipeline without breaking its CI?

⚡ TL;DR: Clay integrates Claygent and Sculptor agents with a rigorous eval loop, uses a data lake for context, and leverages LangSmith for CI‑grade testing.

The Simple Version

Clay runs two main agents: Claygent handles web research and data lookup, while Sculptor powers go‑to‑market outreach.

How It Actually Works

Design of the Agent Stack

The stack layers a web‑search module, a first‑party data connector, and a scaling engine.

🎯 Think of it this way: It’s like a librarian (Claygent) fetching facts and a sales rep (Sculptor) using those facts to close deals.

Eval Philosophy

Four quadrants guide local dev, CI, production metrics, and offline review.

🎯 Think of it this way: Imagine checking a car’s engine, brakes, steering, and interior all before a race.

Data Lake Integration

A unified data lake gives agents instant access to Clay’s internal datasets.

🎯 Think of it this way: It’s like giving every chef a shared pantry instead of hunting for ingredients.

LangSmith Integration

LangSmith tracks runs, surfaces A/B metrics, and closes the loop between online performance and offline evals.

Technical Deep Dive

Evals run on every commit, with latency, hallucination, and safety scores feeding into CI pipelines.

When a context breakthrough from Fable expanded token limits, agents could ingest full conversation histories, dramatically improving response relevance.

Scaling decisions prioritized a data lake over a traditional warehouse to keep latency low and query flexibility high.

Why This Matters

Agents that can safely access internal data open new product experiences—real‑time research, personalized demos, and automated workflows.

By publishing its eval stack, Clay sets a benchmark for other AI startups on how to ship agent updates without sacrificing reliability.

Developers can adopt similar patterns using LangChain, LangSmith, and data‑lake architectures to accelerate their own agent pipelines.

Key Takeaways

Agent safety hinges on a four‑quadrant eval framework.
Data lakes give agents first‑class access to internal knowledge.
LangSmith provides CI‑grade observability and feedback loops.
Large‑context breakthroughs enable richer, more accurate agent responses.
Scaling agents requires tight coupling between evaluation, data, and deployment pipelines.

Clay’s 300M‑Run Agent Stack: Eval‑First Scaling with LangSmith