Stop Breaking Your AI Agents: A Guide to Evals for Managed…

Learn how to use Harbor and LangSmith to catch regressions and systematically improve your AI agents using Managed Deep Agents evals.

Ever added a new tool to your AI agent only to find out it suddenly forgot how to do everything else? That's the nightmare of AI regressions, and it's why 'evals' are the most important part of your agent's lifecycle.

⚡ TL;DR: Managed Deep Agents use Harbor to run isolated evaluations in fresh containers. This allows developers to catch regressions, 'hill climb' toward better performance, and monitor everything via LangSmith.

The Simple Version

In traditional software, we have unit tests to make sure a change in line 10 doesn't break line 100. AI agents are trickier because they are non-deterministic—they don't always give the same answer twice.

Evals are essentially a set of rigorous tests that check if your agent can still complete specific tasks correctly after you've made changes.

🎯 Think of it this way: Imagine you're training a dog. Every time you teach it a new trick (like 'roll over'), you still ask it to 'sit' and 'stay' just to make sure it hasn't forgotten the basics while learning the new stuff.

How It Actually Works

Managed Deep Agents utilize a system called Harbor to make testing reliable and repeatable.

The Anatomy of an Eval

An evaluation isn't just one test; it's a three-part stack:

Environment: A fresh, isolated container where the agent operates so there's no 'leftover' data from previous runs.
Job: The specific task or goal the agent is asked to achieve.
Check: The criteria used to determine if the agent actually succeeded.

🎯 Think of it this way: The Environment is the clean classroom, the Job is the exam paper, and the Check is the answer key the teacher uses to grade it.

Scaffolding and Automation

You don't have to write every test by hand. Using mda evals init, you can scaffold your evaluation suite quickly.

You can even hand this process off to a coding agent, like Claude Code, to generate the test cases based on your requirements.

Technical Deep Dive

For the developers, the magic lies in the integration between Harbor, LangSmith, and Pytest.

Harbor ensures state isolation. By spinning up a fresh container for every eval, you eliminate 'flaky tests' caused by cached data or previous agent interactions.

The workflow integrates directly with LangSmith, where you can track traces and rewards. This allows you to see exactly where an agent deviated from the expected path during a failed eval.

For those who prefer standard tooling, the system supports pytest. This means you can write readable, Pythonic tests that fit right into your existing CI/CD pipeline for nightly regression checks.

Why This Matters

As agents move from 'cool demo' to 'production tool,' reliability becomes the only metric that matters. A research assistant that works 90% of the time is a liability; one that works 99.9% of the time is a product.

Without a systematic way to 'hill climb' (incrementally improving performance), you're just guessing if your prompts are getting better or worse.

Implementing these evals transforms AI development from an art of 'vibes' into a rigorous engineering discipline.

Key Takeaways

Regressions are inevitable: Adding new tools often breaks old functionality in LLMs.
Isolation is key: Harbor uses containers to ensure every test starts from a clean slate.
The Eval Trifecta: Every test needs a defined Environment, Job, and Check.
Automated Scaffolding: Use mda evals init to jumpstart your testing suite.
Observability: LangSmith provides the traces needed to debug why an eval failed.
CI/CD Integration: Run evals nightly to ensure your agent doesn't regress before shipping to users.

Stop Breaking Your AI Agents: A Guide to Evals for Managed Deep Agents