A new benchmark called Real-SWE tests frontier AI models on something they’ve never seen before: private production codebases licensed from real companies. The release, which drew strong attention on Hacker News, targets a question the industry keeps dancing around. Can a coding agent actually do the job of a software engineer, not on a toy repo, but inside a business that ships product every week?
The answer, according to the team behind Real-SWE, depends on where you look. Public benchmarks measure one thing. Enterprise reality measures another.
What Real-SWE actually is
Every task in Real-SWE comes from a private codebase that the researchers licensed directly from an operating company. These aren’t synthetic puzzles or expert-written exercises. As the Hacker News post puts it, they’re “problems their engineers work on, with all the context and complexity that comes with an existing product.”
The team screened codebases for real usage, strong engineering teams, and demanding production workloads. Sample sources include:
- A social events app competing with Luma and Partiful, with 200K+ users and a top 100 App Store ranking
- A consumer fintech platform that processes 100K+ bank statements
- Enterprise AI sales platforms handling complex business workflows
The tasks reflect that mix. Billing logic. Tax calculations. Customer migrations. Changes that ripple across multiple services and carry real business consequences if you get them wrong.
Why private code changes the game
This is the part that matters most. Every popular coding benchmark (SWE-bench, Terminal Bench, DeepSWE) draws from public repositories. That means the code, the issues, and often the fixes exist somewhere on the internet. Models trained on massive web crawls may have already seen the answer, or at least something close to it.
Real-SWE closes that door. The researchers note that agents “must navigate proprietary systems whose code and solutions aren’t available on the public internet.” No memorization shortcut. No familiar open-source patterns to lean on. Just an unfamiliar codebase with its own conventions, and a task that has to fit inside them.
What stands out here is the emphasis on company-specific complexity. Every engineering team has house rules: how they structure services, name things, handle errors, write tests. A correct change that ignores those rules isn’t correct in practice. Real-SWE grades agents on whether their work fits the existing system, not just whether it passes a verifier.
How the evaluation works
The methodology has three notable design choices, as detailed in the Hacker News release:
- Underspecified prompts, on purpose. Tasks describe the change needed but leave implementation details for the agent to discover in the codebase and surrounding tools. The team calibrated this to be “about par with DeepSWE and Terminal Bench,” so instructions are vague enough to require investigation but never omit something the verifier demands.
- Native harnesses. Instead of testing models in isolation, Real-SWE evaluates model-and-harness combinations. That mirrors how enterprise engineers actually work with tools like Claude Code, Codex, or Cursor. The harness is part of the product, so it’s part of the score.
- Cross-functional tasks. A single change often spans multiple parts of an application. Agents need to understand existing business logic and keep the surrounding system working, not just satisfy a narrow test.
What the release doesn’t tell us yet
The announcement describes the benchmark design in detail but doesn’t publish a leaderboard or specific pass rates in the excerpt shared. The team says these tasks “challenge today’s frontier models,” which reads as a polite way of saying scores are lower than public benchmarks suggest. Until numbers land, treat that as a directional signal, not a verdict.
There’s also an inherent tension in the approach. Private codebases can’t be fully open-sourced, so independent reproduction is harder than with SWE-bench. Readers should watch for how the team handles transparency and contamination as the benchmark ages.
Why practitioners should care
If you’re deciding whether to trust an agent with production work, this is the benchmark to track. Public leaderboards tell you how well a model handles open-source patterns. Real-SWE tells you how it handles your kind of mess: legacy conventions, undocumented business logic, and changes where a mistake costs money.
A few practical takeaways:
- Stop reading SWE-bench scores as enterprise readiness. They measure something narrower than the marketing implies.
- Evaluate the harness, not just the model. The same model behaves differently in different agent tools. Test the combination you’ll actually deploy.
- Build your own mini Real-SWE. Pull five to ten recent tickets from your private repo, hand them to an agent with your normal tooling, and grade the results against your conventions. That’s a better signal than any public number.
The broader trend is clear. As models saturate public benchmarks, the industry needs evaluation that mirrors the work people actually pay for. Real-SWE is an early move in that direction, and the first published leaderboard will be worth a close look.