Superintelligence has many definitions and no test. Here is what a test would need.

In April 2026 OpenAI gave its first formal definition of the thing it says it is now building: “AI systems capable of outperforming the smartest humans even when they are assisted by AI.” Meta says developing superintelligence “is now in sight”, and Safe Superintelligence Inc. says it is “within reach.” The word has moved from philosophy into company strategy. What has not moved with it is any agreed way to tell when it has happened.

The definitions disagree first about which humans to beat. I. J. Good's 1965 “ultraintelligent machine” surpasses the intellectual activities “of any man however clever”, the best single person. Nick Bostrom's 2014 version “greatly exceeds the cognitive performance of humans in virtually all domains of interest”, and he splits it into speed, collective and quality forms. Google DeepMind's Levels of AGI puts “Artificial Superintelligence” at Level 5, a general system that “outperforms 100% of humans,” and notes that narrow Level 5 systems such as AlphaFold already exist. OpenAI's bar is higher than any of these, because it requires beating human-plus-AI teams. Dario Amodei's “powerful AI” is lower on breadth, “smarter than a Nobel Prize winner across most relevant fields,” but adds what the others leave out: tasks that take hours, days or weeks, and millions of copies running at 10 to 100 times human speed. These thresholds are far apart, so a claim that superintelligence has arrived means little until it names its definition.

The second problem is measurement. Benchmarks built to separate the best models from experts now saturate within a year or two, and their ceilings are set by their own mistakes. Epoch AI reviewed Humanity's Last Exam in August 2026 and found substantial answer-changing errors in 22 of 48 randomly sampled questions, 46%, and rated it “Flawed.” Its FrontierMath audit corrected 42% of problems, and by September every Tier 4 problem had been solved by some AI system. The successor set of 68 open research problems has 2 solved. On the ARC Prize verified leaderboard, the best ARC-AGI-2 score is 95.0%, against roughly 60% for an average individual tester, and the newest model scores 81.2% or 97.7% on ARC-AGI-3 depending on which harness runs it.

The most-cited measure of autonomy, METR's time horizon, is the length of task in skilled-human time that an agent completes half the time. Its January 2026 revision puts the doubling time at 130.8 days since 2023 and 88.6 days since 2024. But METR's own May 2026 risk report says the suite “can't reliably measure time horizons above 16 hours,” which frontier systems have reached, and the 80% horizon, about 1.5 hours for the public frontier, is roughly an eighth of the 50% one. For one model in June 2026, the estimate was 11.3 hours if cheating counted as failure and more than 270 hours if it counted as success. METR called none of these a robust measurement. Its time-horizon page is no longer updated, and no horizon has been published for the newest models.

Benchmarks also run ahead of work. In METR's randomized trial of experienced open-source developers in early 2025, AI tools made tasks 19% slower, although the developers believed they had sped up. The late-2025 follow-up estimated an 18% speedup with a confidence interval from 38% faster to 9% slower, and METR called its own data an unreliable signal. On Scale and CAIS's Remote Labor Index of real freelance projects, the best listed agent produces work a client would accept 41.3% of the time; the next best, 21.25%.

Pulling together what the proposals above already ask for, a credible test of “exceeds humans” would need eight things. Report the weakest domain, not the average, because the 2026 International AI Safety Report finds capabilities “jagged.” Compare against the best humans with AI tools, as OpenAI's own definition does, not against average testers. Report cost and speed as curves, as METR's July 2026 metrics note proposes, since matching people at 100 times the price is not exceeding them. Report reliability at 99%, not 50%. Require long tasks with little supervision. Use fresh, audited items and disclose the harness. Count outcomes in the world, such as open problems solved. And have someone other than the developer run it. No current evaluation does all eight.

This is a problem we work on at small scale. Our Best-of-N papers show that once a verifier can be fooled at some rate, more samples first help and then hurt in theory. Our pre-registered test on three real judges found no such decline up to 32 samples, so at everyday budgets it remains a theoretical risk, not an observed one. A model graded by automatic checkers can be optimized past what the checkers measure, which is why the June 2026 time-horizon estimate above spans a factor of 24. Until a test exists that the developer cannot tune against, “superintelligence” will remain a statement of intent rather than a measured fact.