90% of US working hours are spent on tasks not covered well by AI benchmarks

To measure benchmark coverage, we start with a list of 29 benchmarks selected for their quality and economic relevance. (The full list and selection criteria are in the Benchmark Index tab). In selecting benchmarks for this list, we prioritized:

Following work by Wang et al., we then ask an LLM judge to tag each benchmark prompt with as many ONET tasks as it thinks correspond to that prompt. We validate our LLM tagging with human expert tags on Mercor’s APEX benchmarks, a subset of this list of 29. On the Benchmark Map, an ONET task counts as “covered” if our LLM judge has mapped it to at least 5 prompts from our 29 benchmarks.

To estimate the total share of working hours Americans spend on tasks covered by benchmarks on this list, we combine our coverage estimates with Hatgis-Kessell et al.’s LLM-derived estimates of the working time share of each O*NET task.

A more detailed account of our methodology can be found in the Methodology tab.

Our estimates of total economic coverage, and of patterns in coverage, all come from the initial list of 29 benchmarks we chose. This is not a complete or perfectly representative sample of the broader universe of unsaturated, public, agentic economic benchmarks. Any incomplete list will necessarily bias down estimates of economic coverage, and it is difficult to bound that bias. We want to stress, therefore, that these results are provisional. We will update them as we expand the initial benchmark list.

We also capped the number of included benchmarks in each job family in order to maximize breadth of coverage. We think this makes our estimate of economy-wide benchmark coverage more accurate, at the cost of biasing down our estimates of coverage in the most-benchmarked job families (e.g. software engineering).

By selecting for unsaturated benchmarks, we risk understating benchmark coverage on O*NET tasks which are easier for LLMs to complete. On the other hand, including saturated benchmarks would risk overstating benchmark coverage if saturation is downstream of lower task realism. The latter claim seems likely, given that benchmarks often overstate LLM performance in real-world deployments. We think that the bias from including saturated benchmarks would be greater than the bias from excluding them, but neither option is ideal.

Finally, while we outline our benchmark selection criteria in the Benchmark Index tab, several criteria (like “Economic Realism”) are difficult to formalize and introduce some subjectivity in our choices.

The benchmarks in our index have relatively deep and narrow coverage of the overall economy. They disproportionately focus on the economic tasks most exposed to AI automation (as measured by Eloundou et al.), with O*NET tasks rated as “exposed” about 6x more likely to be represented on benchmarks than tasks rated “unexposed”. This makes sense: benchmarks are designed to measure AI progress, and AI progress is most plausible on the most AI-exposed tasks. This explains why benchmarks are full of tasks in Software Engineering and Financial Analysis while sparse on tasks in Truck Driving and Gas Plant Operation.

But even after controlling for AI-exposure, O*NET tasks from occupations with larger wage bills are significantly overrepresented among benchmark prompts: