Imagine playing a game of 'Telephone' where the players are AI agents. By the time a rumor about a mango sale reaches the fifth agent, the original source is forgotten, and a 'maybe' has magically transformed into an absolute fact.
This is the core challenge of long-horizon agent behavior: as state persists over time, social consistency decays. Erina Karati, a former engineer at Microsoft and Supercell, reveals how she and Arunachalam Manikandan tackled this using a framework called Project Paradox.
The Simple Version
Most people try to fix AI agents by writing better prompts. This is like giving an actor a better script for a single scene; it works great for five minutes, but it doesn't help them remember their character's motivations over a whole movie.
Karati argues that for agents to behave consistently over long periods, we need to stop 'prompting' and start 'experimenting.' Instead of guessing which words will work, they built a system that runs thousands of simulations, sees where the agents fail, and tweaks the underlying rules until the behavior improves.
How It Actually Works
The Stateful Architecture of Project Paradox
Project Paradox wasn't just a prompt; it was a modular framework designed to give agents a sense of 'self' and 'history.' To achieve this, Karati implemented a stateful architecture with four key pillars.
First, each agent has its own memory namespace backed by RAG (Retrieval-Augmented Generation), ensuring that memories don't 'bleed' between characters. Second, they tracked emotion as a small vector containing values for joy, sadness, fear, anger, and disgust.
Third, they implemented belief scores—essentially a trust matrix—that the LLM updates after every interaction. Finally, they used importance scores to decide what to keep. As Karati explains, "if someone was murdered a few days ago, you definitely remember that," whereas dinner details are discarded.
The Breakdown: The Mango Rumor
While this worked for short interactions, like planning a picnic, it failed over long horizons. In one test, a rumor about a mango sale spread through the village, but the agents eventually lost track of who started the rumor or began treating a 'might' as a certainty.
This is a classic problem in multi-agent systems known as state drift. Because there is no central database—only agents talking to agents—information degrades. As Karati puts it, "the system may remember the rough topic but lose the source of the topic."
The Auto Research Loop
To fix this, the team wrapped the village in an 'auto research' layer, inspired by work from Andrej Karpathy. This layer acts as a meta-system outside the game world that monitors everything the agents do.
The loop follows a strict cycle: define a controlled scenario, run the simulation, collect structured traces (conversations, memory writes, belief updates), and score the results. If the score is low, the system proposes a small change to the policy surface and reruns the test.
Technical Deep Dive
The brilliance of the Project Paradox approach lies in the Balanced Scorecard and the Constrained Policy Surface. In many AI systems, optimizing for a single metric leads to 'reward hacking,' where the model finds a shortcut that increases the score without actually improving the behavior.
To prevent this, Karati used a multi-dimensional scorecard. For information diffusion, they measured reach (how many agents knew the fact) and provenance (source retention). For rumors, they measured uncertainty preservation to ensure a 'maybe' didn't become a 'fact.' They also tracked containment to ensure private information wasn't leaked.
For context, this is similar to how RLHF (Reinforcement Learning from Human Feedback) uses a reward model to guide a LLM, but instead of human preferences, it uses objective behavioral traces from a simulation.
Crucially, the auto research layer cannot rewrite the entire codebase. It can only edit a small, frozen policy surface. This includes specific protocols like memory writing policies, retrieval rules, and replanning triggers. By limiting the 'editable surface,' the engineers prevent the LLM from simply gaming the evaluation metrics.
The process functions as a ratchet mechanism. A change is only kept if the scorecard improves and the guardrails hold; otherwise, the system performs a rollback. This is critical because, in complex social systems, fixing one bug (like slow information spread) often creates another (like leaking private secrets).
Why This Matters
While Project Paradox was built for a game village, the implications extend to any AI system that maintains state over time. Most current AI agents are 'stateless' or rely on a simple chat history, which inevitably collapses as the conversation grows.
For example, customer support agents must know which policy update is the most recent and whether it supersedes an older one. Personal assistants need to remember commitments and handle corrections without losing the original context. Coding agents must maintain long-running context across multiple files, teammates, and shifting requirements.
As Karati concludes, "long horizon agent behavior needs scenario suits." The shift from 'vibes-based' evaluation (watching one nice demo) to rigorous, scenario-based experimentation is the only way to build reliable, stateful AI that doesn't forget who it is or where its information came from.