AI benchmarks have a blind spot, and it’s a big one. About 90% of US working hours go to tasks that today’s AI benchmarks don’t measure well, according to a new analysis that drew attention on Hacker News. The researchers say the number is provisional. Even so, if it’s roughly right, the scores you see on leaderboards tell you very little about most of what people actually do at work.
What the researchers did
The team started with 29 benchmarks. They picked ones that fit three rules:
- Unsaturated: today’s top models haven’t maxed them out yet.
- Public: anyone can look at them and run them.
- Agentic and economic: they test AI doing real multi-step work, not trivia or puzzles.
They then matched those benchmarks to O*NET, the US government’s big database of job tasks. O*NET splits occupations into specific tasks, which lets researchers weight each task by the hours workers actually spend on it. Finally, they asked how much of that total working time the benchmarks really cover.
The answer: around 10%.
The limitations, which the authors put up front
What stands out here is how openly the researchers lay out the weak spots in their own method. That’s rare, and it helps you read the result correctly. Here’s what they flag:
- The sample is incomplete. Twenty-nine benchmarks aren’t the whole universe. The authors note that “any incomplete list will necessarily bias down estimates of economic coverage, and it is difficult to bound that bias.” So the real coverage is probably somewhat higher than 10%.
- They capped benchmarks per job family. This was done to get breadth across the economy. The downside is that heavily benchmarked fields, like software engineering, look less covered than they really are.
- They left out saturated benchmarks. That could understate coverage on tasks LLMs already find easy. But the team argues that adding them would cause a bigger error, since saturation often comes from tests that aren’t realistic. In their words, “benchmarks often overstate LLM performance in real-world deployments.”
- Some judgment calls are subjective. Criteria like “Economic Realism” are hard to pin down, so different researchers might have picked different benchmarks.
The team says it’ll update the figures as the benchmark list grows.
Why this matters
This is significant because benchmarks drive so much of the AI industry. Labs use them to market models. Buyers use them to choose vendors. Investors use them to judge progress. If the tests cluster around a small slice of work, mostly coding, math, and some office tasks, then “AI is getting better at everything” is a much shakier claim than it sounds.
The saturation point deserves extra attention. Many people take a maxed-out benchmark as proof that a model has mastered a skill. The researchers suggest the opposite often happens: tests get saturated because they’re too clean and too easy, not because the models are ready for messy real-world work. Anyone who has watched a model ace a demo and then stumble in production will recognize this.
There’s a supply problem too. Benchmark builders go where the data is easy to get and the grading is simple. Software is great for that, because code either passes the tests or it doesn’t. Nursing, skilled trades, sales calls, and managing people are much harder to score, so they get far less attention. The gaps follow convenience, not economic importance.
What practitioners should do with this
If you’re deciding where to put AI in your business, here’s how I’d use these findings:
- Don’t read leaderboard scores as job readiness. A top score on a coding benchmark says almost nothing about how a model will handle your customer support queue or procurement workflow.
- Build your own evals. Take 20 to 50 real examples from your workflow and test candidate models on them. That small, unglamorous test set will be worth more than any public ranking.
- Be extra skeptical of saturated tests. When a vendor points to a near-perfect score, ask how realistic the test was.
- Look for the gaps. If your industry has no serious benchmark, that’s a risk to you and a possible opening for whoever builds a good one.
What comes next
The 90% figure will probably come down as the researchers add more benchmarks, and they say so themselves. But even if real coverage turns out to be two or three times higher, most of the economy would still go unmeasured. The useful takeaway isn’t the exact number. It’s that we’re judging AI’s economic impact with a test suite that skips most of the exam. Teams that measure AI on their own real work will make better calls than teams that trust the leaderboard. You can find the full methodology and benchmark index at the original source.