Knowing the answer to a trivia question is one thing, but fixing a crashed production database is another entirely. Supabase has just shifted the goalposts for AI evaluation by moving away from theoretical tests and toward real-world chaos.
The Simple Version
Most AI benchmarks are like multiple-choice tests in a classroom. They measure if a model has read the right books, but they don't prove the model can actually do the work in a real environment.
Supabase's new evaluation is different. Instead of asking questions, they drop AI agents into actual Supabase projects and give them a 'to-do' list based on real problems users have faced. This includes tasks like building a database from scratch, fixing a broken security rule, or debugging a function that crashed.
How It Actually Works
The core of this benchmark is grounded in reality. Rather than synthetic data, Supabase used actual support tickets and bug reports to create the test cases. This ensures the AI is fighting the same battles that human developers fight every day.
The Performance Leaderboard
The results show a tight race at the top, with some surprising gaps in the middle. According to the data, Claude code with Opus 5.5 and Grok 4.7 both scored 90%.
Following closely was Kimmy K 3 with a score of 87%. Surprisingly, GPT-6 lagged behind the leaders, scoring only 78% on these practical tasks.
The Power of 'Agent Skills'
One of the most revealing parts of the test involved Sonnet 5. Initially, the model scored 78% on building tasks, putting it on par with GPT-6.
However, when the developers provided the model with the 'Supabase agent skill,' its performance skyrocketed. As the video notes, the score "jumped to 100%" for the exact same model. This suggests that the limitation wasn't the model's intelligence, but its access to specific, optimized tools and context.
Technical Deep Dive
To understand why this benchmark matters, we have to look at the difference between General LLM Reasoning and Agentic Tool-Use. When a model like Opus 5.5 or GPT-6 attempts to 'fix a security rule,' it isn't just generating text; it is interacting with a live environment.
The Complexity of Supabase Tasks
Fixing a 'broken security rule' in Supabase specifically refers to Row Level Security (RLS). RLS is a PostgreSQL feature that allows developers to define policies on who can see or modify specific rows in a table.
Writing a correct RLS policy requires the AI to understand the relationship between the authenticated user (via JWTs) and the data schema. A small syntax error or a logic flaw in a policy can either lock everyone out of the database or, worse, expose private data to the public.
Debugging Edge Functions
The 'crashed function' tasks likely involve Supabase Edge Functions, which run on Deno. To debug these, an agent must be able to read logs, identify the stack trace, and modify TypeScript code to handle edge cases or API failures.
This requires a loop of Observe → Plan → Act. The agent observes the error, plans a fix, applies the code change, and tests the result. This is significantly more complex than the single-turn prompt-and-response pattern used in standard benchmarks.
What is an 'Agent Skill'?
The jump from 78% to 100% for Sonnet 5 highlights the importance of Function Calling and RAG (Retrieval-Augmented Generation). An 'agent skill' is essentially a set of well-defined tool definitions (JSON schemas) and a specialized knowledge base.
For context, when a model has a specific skill, it doesn't rely on its training data (which might be outdated). Instead, it uses the skill to query the latest Supabase API documentation or execute specific CLI commands. This removes the 'hallucination' factor, as the model is guided by the actual constraints of the system.
Why This Matters
This shift toward 'real-world' evals marks a transition in the AI industry. We are moving away from the era of the 'Chatbot' and into the era of the 'Agent.' The goal is no longer just to have a conversation, but to have a tool that can autonomously manage infrastructure.
As the speaker puts it, "the whole point of this benchmark is for the models to actually become benchmarks because then they get better at using Supabase." By creating a standardized way to measure success on real tickets, Supabase is effectively creating a feedback loop that forces model providers to optimize for their platform.
This ecosystem is already massive, with companies like GitHub, Lovable, Figma, and LangChain building on Supabase. When AI agents become 100% reliable at managing these backends, the barrier to building complex, scalable applications drops to nearly zero.