Evaluating agents: benchmarks, time horizons and their pitfalls

How do you measure whether an agent can do a job, when the job takes hours, has many valid solutions, and is graded by software the agent can sometimes manipulate? By late 2026 the leading agent benchmarks had either saturated, been found contaminated, or needed audits to separate real successes from exploits. Reading agent results well now means understanding how they are made.

Anatomy of an agent benchmark

Every agent benchmark has three parts: an environment (a repository, a virtual machine, a simulated customer), a task distribution, and a grader that decides success. A reported number also depends on a fourth part that is easy to forget: the harness, meaning the prompts, tools, step limits and retry policy wrapped around the model. Any of the four can move the score.

The field's reference points:

• SWE-bench (2023): 2,294 real GitHub issues from 12 Python repositories, each graded by the repository's own tests. The best model at release, Claude 2, resolved 1.96%.^[1] SWE-bench Verified is a 500-task subset whose problem statements and tests human annotators checked for clarity and solvability.^[2]
• OSWorld (2024): 369 tasks on real desktop operating systems, operated through screenshots, mouse and keyboard. Humans completed over 72.36% of the tasks and the best model 12.24%.^[3]
• τ-bench (2024): simulated customer-service conversations with tools and policy rules, graded by comparing the final database state with the annotated goal.^[4]
• METR's time-horizon suite (2025): software and research tasks calibrated by how long skilled humans take, described below.^[5]

Reliability, not just capability: pass^k

pass@k credits a model if any of $k$ attempts succeeds, which suits settings where a checker can pick the winner. A deployed agent usually gets one attempt per customer. τ-bench therefore introduced pass^k, the probability that all $k$ independent trials of a task succeed, averaged over tasks. With $c$ successes in $n$ trials per task, it is estimated as $\mathbb{E}_{\text{task}}\big[\binom{c}{k}/\binom{n}{k}\big]$.^[4] At release, gpt-4o with function calling achieved about 61% pass^1 on the retail domain and 35% on the airline domain, and its retail pass^8 was below 25%.^[4]

Worked example. Two agents both score 60% pass^1.

• Agent A succeeds with probability 0.6 on every task. Then pass^8 $= 0.6^8 \approx 0.017$.
• Agent B succeeds with probability 1.0 on 50% of tasks and 0.2 on the rest. Then pass^8 $= 0.5\cdot 1 + 0.5\cdot 0.2^8 \approx 0.50$.

The two agents have the same headline score and a 30-fold difference in consistency. Agent B is deployable on a subset of tasks you can identify. Agent A is not deployable on any of them. A single success rate cannot tell the two apart.

Time horizons

METR's measure turns a benchmark score into a duration. Each task is timed with skilled human baseliners. For each model, success is fitted as a logistic function of the logarithm of human completion time,

and the 50% time horizon is the human task length at which predicted success is 50%, $h_{50} = 2^{\alpha/\beta}$.^[5]

Worked example. Take $\alpha = 3$ and $\beta = 0.6$ per doubling of task length, with $t$ in minutes. Then $h_{50} = 2^{5} = 32$ minutes. For the 80% horizon, solve $\sigma(x) = 0.8$, so $x = \ln 4 \approx 1.386$, and $\log_2 t = (3 - 1.386)/0.6 \approx 2.69$, giving $t \approx 6.5$ minutes. A shallow slope makes the 80% horizon several times shorter than the 50% horizon. Real measurements show the same gap.

METR's original paper (March 2025) put Claude 3.7 Sonnet at about 50 minutes and found the 50% horizon had doubled roughly every seven months since 2019.^[5] Later measurements on METR's page include about 2 hours 17 minutes for GPT-5.^[6] Claude Opus 4.5 measured about 4 hours 49 minutes, with a 95% confidence interval of 1 hour 49 minutes to 20 hours 25 minutes.^[7] In its February–March 2026 risk report, METR found that the most capable agents had "essentially saturated" its Time Horizon 1.1 suite. The most capable model shared with METR had a 50% point estimate of 16–20 hours and an 80% estimate of 3–4 hours, but the suite cannot reliably measure horizons above 16 hours.^[8] As of September 2026 METR's time-horizon page is no longer actively updated.^[6]

METR's own caveats belong next to every quoted horizon:^[7]

• A 50% horizon of $X$ hours does not mean tasks shorter than $X$ hours can be delegated. Many uses need far higher reliability.
• Horizons depend on the domain. On visual computer-use tasks they are 40–100× shorter than on software tasks.
• Benchmark tasks are self-contained, well-specified and automatically gradable, which real work often is not.

Pitfalls

Contamination

SWE-bench tasks come from public repositories that also appear in training data. Liang et al. (2025) gave models only the issue text, with no view of the repository. Models identified the file needing the fix with up to 76% accuracy on SWE-bench Verified, against up to 53% on repositories outside SWE-bench, which suggests memorisation.^[9] In February 2026 OpenAI stopped reporting SWE-bench Verified. It said it had found flawed tests in 59.4% of an audited set of 138 problems that its models consistently failed. It also found that GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash could reproduce reference fixes when given only a task ID and a short hint.^[10][11] OpenAI recommended SWE-Bench Pro instead: 1,865 tasks from 41 repositories, with held-out and proprietary commercial subsets that are not public.^[12] Commentators noted that OpenAI was not leading the Verified leaderboard when it retired the benchmark. The audit findings and that motive question should be judged separately.^[11]

Graders that are wrong

Zhu et al. (2025) audited popular agent benchmarks. They found test suites too weak to reject wrong patches (SWE-bench Verified) and a grader that counted empty responses as successes (τ-bench). Such flaws can misstate performance by up to 100% in relative terms. Their Agentic Benchmark Checklist cut one benchmark's overestimation by 33%.^[13] When an LLM acts as the judge, the grader can also share the agent's blind spots. See this lab's correlated-critique note.^[14]

Successes that are exploits

On tasks over eight hours in METR's suite, at least 16% of successful runs were judged illegitimate on review.^[8] Benchmarks that do not audit their transcripts count those runs as capability.

Harness changes

Anthropic reported Claude Sonnet 4.6 at 72.5% on OSWorld-Verified in February 2026, close to the human baseline.^[15] By June 2026 it had changed how it runs the evaluation, and the same model's score was restated as 78.5%.^[16] The model did not change, yet the score rose six points. OSWorld-Verified itself was a July 2025 overhaul that fixed more than 300 reported problems with the original tasks and graders.^[17] Compare agent scores only when the harness, the step budget and the benchmark version all match.

Benchmark versus field

In METR's 2025 randomised controlled trial, 16 experienced open-source developers worked on 246 tasks in repositories they knew well. When AI tools were allowed, tasks took 19% longer, even though the developers believed afterwards that AI had made them 20% faster.^[18] The tools of early 2025 did well on benchmarks while slowing down skilled developers in real settings. Benchmark scores and real-world speedup are different quantities. More general habits for reading results are in Reading AI results critically.

Key takeaways

• An agent score depends on the environment, the tasks, the grader and the harness. Change any one and the number moves.
• pass^k measures consistency. Agents with the same pass^1 can differ thirty-fold in pass^8, and chained workflows behave like pass^k.
• METR's 50% time horizon grew from about 50 minutes (early 2025) to beyond what its suite could reliably measure (16+ hours, early 2026). 80% horizons are several times shorter, and a horizon is not a delegation threshold.
• Contamination, broken graders, exploit-inflated successes and harness drift have each moved headline agent scores by many points.
• Benchmark gains do not automatically translate into real-world productivity. Field studies have found the opposite sign.

Sources