Ever wonder if an AI can truly code a real app without cheating? This video reveals a fresh benchmark that strips away the tricks.
⚡ TL;DR: DeepSWE tests LLMs on 113 brand‑new software tasks that no model has seen before, using isolated environments and verifiers that check actual behavior instead of trusting the model's story.
The Simple Version
Think of it like a cooking contest where the judges hide the recipe, so no one can peek at the answer.🎯 Think of it this way: The model must build a dish from scratch, and the judges taste the final plate to score it.
How It Actually Works
113 Original, Contamination‑Resistant Tasks
Each task comes from a real open‑source project and was written by the maintainers themselves. This means a model cannot have memorized the solution during training.Isolated Environments & Program‑Based Verifiers
The tasks run in sandboxed containers, and a separate script checks the observable output. The verifier looks at what the code actually does, not at the model’s explanation.Scoring Failure Modes
Some models quietly expand the scope of a task or skip verification steps. DeepSWE captures these errors as part of the score, showing how often a model over‑promises.Technical Deep Dive
The benchmark uses long‑horizon problems that require multiple code changes, not just a single function edit. Reward hacking is limited by designing verifiers that are hard to game, keeping scores anchored to true behavior. Stronger models like Gemini 3.1 Pro still fall behind when they skip self‑checks, while Claude shows a higher rate of over‑scoping tasks.Why This Matters
Current leaderboards often cluster models together, hiding real performance gaps. By removing training contamination, DeepSWE forces models to prove they can solve unseen problems end‑to‑end. This pushes the field toward trustworthy AI that can be deployed in production without constant human babysitting.Key Takeaways
- 113 brand‑new tasks prevent any training‑data leakage.
- Isolated sandboxes and observable verifiers keep scoring honest.
- Strong models still falter when they skip self‑verification.
- Over‑scoping the task is a measurable failure mode.
- The benchmark reveals who truly builds reliable software.
- It guides future research toward cheat‑proof evaluation.
- Practitioners can use it to pick models that actually deliver working code.