Ever wonder how a bank with 135 million customers can roll out AI agents 20× faster? The answer lies in smart simulations.
The Simple Version
Think of an AI agent like a virtual teller. Instead of waiting for real customers to test it, you create fake chats that mimic real ones.
How It Actually Works
Grounded Simulations
Snowglobe builds simulations that include account details, tone, and intent. These are called grounded simulations.
Eval Loop
The system runs the agent, observes the output, creates new simulations, and repeats. This loop finishes in minutes, not weeks.
Technical Deep Dive
Snowglobe uses a language model to generate multi‑turn dialogues that preserve state across turns. The model pulls from Nubank’s customer data schema, ensuring each simulated turn references the correct account balance or product.
Metrics such as intent alignment and tone match are computed automatically, giving a score that correlates with human review at about 80 % accuracy.
Why This Matters
Faster evals mean product teams can iterate on AI agents without bottlenecks. The result is higher customer satisfaction and the ability to safely launch new features at scale.
Enterprises can now test open‑source models and custom variants with cheap, repeatable simulations, accelerating innovation across the board.
Key Takeaways
- Simulations replace slow, hand‑curated test sets.
- Grounded simulations embed real account context.
- Eval scores from simulations match human judgment ~80 %.
- Shipping speed improved 20× for Nubank.
- Human review workload dropped dramatically.
- Teams can freely experiment with new model variants.
- Cheap, fast feedback loops boost AI reliability.