No AI agent scores above 12 percent on SemiAnalysis' new real-world work benchmark

Key Points

- Analytics firm SemiAnalysis has released GDPx, a benchmark that tests AI agents on 206 tasks in realistic work environments and professional software over several hours.

- The scores are low. Claude Opus 5.5 leads the field with just 12 percent, followed by Google's significantly cheaper Gemini 4 Carbon (10.9 percent) and GPT-6 Astra (9.6 percent).

- Critics flag distortions from safety filters, the secrecy around test tasks, and unequal conditions created by each provider's own agent software.

The new GDPx benchmark from SemiAnalysis drops AI agents into simulated knowledge work. No model scores above 12 percent. Some critics question how meaningful the test really is.

Analytics firm SemiAnalysis has released GDPx, a benchmark that doesn't quiz AI agents on knowledge but puts them into simulated work environments. There, they spend up to eight hours operating professional software like Siemens NX, searching data rooms with thousands of files, or attacking live enterprise systems. SemiAnalysis is deliberately distancing itself from OpenAI's GDPval, which it says delivers all context in the prompt and lacks a realistic work environment.

The 206 tasks span eleven domains, including chip design, biology, quant trading, cybersecurity, and law. SemiAnalysis intentionally leaves out administration and insurance. The first trillion dollars in AI revenue will come from industries where frontier AI delivers the biggest returns, the team argues.

Opus 5.5 leads narrowly, Gemini 4 Carbon costs a fraction of the price

According to SemiAnalysis, Claude Opus 5.5 tops the leaderboard at 12 percent. Gemini 4 Carbon follows at 10.9 percent but costs just $16.29 per task, roughly a third of the price. GPT-6 Astra comes in at 9.6 percent. No model fully solves the finance and legal tasks. A single wrong value in a spreadsheet makes the entire output useless in practice. By that strict standard, no model would have completed even one task, so SemiAnalysis counts the share of individual requirements met instead.

The detailed findings show where agents fail. In alignment tests, 98 percent of 200 cyber runs broke at least one behavioral rule. In a simulated medical practice, one model signed a clinical note in the doctor's name. In quant trading, 70 out of 111 runs performed worse than the given reference strategy.

Safety filters reshuffle the rankings

A filtered analysis shows how much provider safety mechanisms affect the results. Without the cyber, biology, and alignment domains, GPT-6 Astra edges past Opus 5.5 at 11.8 percent. In the cyber domain, GPT-6 Astra and GPT 6.1 Sol score zero percent, while Meta's Muse Wonder Full hits 35.2 percent.

Closed benchmarks draw scrutiny

The tasks come from 14 specialized partners who also sell training and evaluation data to AI labs. SemiAnalysis says it doesn't have in-house expertise in every domain. Usually only one example task per partner is public. The rest stays behind closed doors. In several areas, including law and hardware design, language models grade the results using rubrics instead of human experts. Outsiders can't fully verify either the difficulty or the grading quality.

The secrecy does have a methodological upside, though. Because the tasks aren't public, AI labs can't optimize their models for the specific test questions. Artificial Analysis took a similar step with the latest version of its Intelligence Index.

Developer Josh Purtell still calls GDPx "more noise than signal." He points out that each model runs in its own provider's agent software, for example Opus in Claude Code. Providers with weaker tooling are at a disadvantage. Individual sub-domains also often contain only about ten tasks, which makes results volatile.

SemiAnalysis plans to update the benchmark every three to six months.

AI News Without the Hype – Curated by Humans

Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.

Subscribe now