Imagine a team where the people labeling data aren’t engineers, yet they’re building the tools they need.
That’s exactly what’s happening at DoorDash.
The Simple Version
Evals stopped being a hidden engineering task and became a shared responsibility.
Non‑engineers can point a coding agent at stable APIs and build the exact UI they need.
Because the platform ships only the API, annotation costs drop dramatically.
How It Actually Works
Trace → Sample → Annotate → Calibrate
The loop starts with tracing a model’s output, sampling a few examples, annotating them, and calibrating a judge against the golden set.
This cycle repeats, keeping quality high without constant engineering hand‑holding.
API‑First Surface
The GenAI platform exposes only stable endpoints for telemetry, datasets, and judge prompts.
Teams build their own UIs on top of those endpoints, rather than waiting for custom tools.
Self‑Serve Judge Calibration UI
A UI shows the original and optimized judge prompts side by side.
Product managers can approve changes without needing a developer.
Technical Deep Dive
Telemetry streams capture every model prediction and user interaction.
Datasets are versioned and stored as immutable snapshots for reproducibility.
Judge prompts are treated as code‑first artifacts, owned by the team that defines the rubric.
Calibration runs automatically through the self‑serve UI, comparing prompt versions on a held‑out set.
Cost tracking shows a 40% reduction in annotation spend after the API‑first shift.
Why This Matters
When evaluation becomes a team sport, quality is set by the people who understand the business need.
Product managers can iterate on rubrics without waiting for engineering queues.
Faster feedback loops mean new features reach customers sooner, and the organization learns continuously.
Key Takeaways
- Evals transition from hidden engineering work to a cross‑functional process.
- API‑first design lets non‑engineers build custom annotation tools.
- Self‑serve judge calibration puts control in the hands of product managers.
- Telemetry and versioned datasets provide the data backbone for continuous improvement.
- Annotation costs can drop dramatically when the loop is automated.
- Org structures that embrace variation see faster learning, not chaos.
- Stable APIs are the foundation for scalable GenAI platforms.