On September 3rd OpenAI introduced GPT-6 Astra with a 99.9% on ARC-AGI-3. The organisation that runs that benchmark published 62.7% the same day. Both numbers are correct and both come from the same model. The difference isn’t the model.
I’ve spent the last few days going through the GPT-6 Astra launch material with a notebook open, and I got stuck on one detail almost nobody has covered.
In the results table OpenAI published there’s a row that reads ARC-AGI-3: 99.9%. Next to it, for context, Claude Opus 5 at 30.2%. The kind of jump that gets half of Twitter typing the letters AGI before lunch.
That same day, ARC Prize — the foundation that designs and administers ARC-AGI — published its own analysis of the model. Their number is 62.7%.
Nobody is lying. Nobody cheated. Both numbers come from running the same model against the same set of problems over the same weekend. What changes between them is something that never shows up in a benchmark table: the harness.
What a harness is
A model doesn’t talk to a benchmark directly. In between sits a layer of plumbing that decides how problems are presented, how much context carries over from one move to the next, what the model is allowed to remember, how many attempts it gets, how results come back. That layer is the harness, and it matters as much as the model does.
ARC Prize ran Astra with two of them.
The first is the Standard harness, which they describe as “the same minimal, provider-neutral interface” for everyone. It “provides all the information required to solve each game, but leaves the model responsible for deciding what to preserve in its visible notes.” And they spell out why that matters: “we believe a future AGI should be able to solve ARC-AGI-3 under these conditions.”
The second is the Provider Adapter, which uses features of OpenAI’s own API and “preserves opaque reasoning state between requests and uses compaction for longer conversations, allowing the model to reuse prior work.” This isn’t a trick — it’s a real product feature, and it’s what a developer would get out of the box building against that API. But it belongs to OpenAI. Anthropic has no equivalent. Neither does Google.
And it’s not a small difference. Across the Public and Semi-Private sets and all reasoning levels, Provider Adapter runs were roughly 3.66× faster by aggregate elapsed time, and used 49% fewer tokens across the 167 game-reasoning pairs both harnesses solved.
Here’s the full table, which is worth reading in its entirety rather than at the headline row:
| Effort | Standard harness | Cost | Provider Adapter | Cost |
|---|---|---|---|---|
| max | 62.7% | $26,098 | 98.6% | $17,332 |
| xhigh | 59.3% | $37,317 | 98.4% | $18,147 |
| high | 54.8% | $40,705 | 99.9% | $18,817 |
| medium | 38.6% | $48,090 | 98.4% | $19,285 |
| low | 17.5% | $38,166 | 98.0% | $21,298 |
| none | 35.2% | $49,791 | 96.7% | $23,457 |
Three things jump out.
First, the headline 99.9% is precisely the high row of the Provider Adapter. Under the Standard harness, that same row gives 54.8%.
Second, with the Provider Adapter the effort level barely matters: three points separate best from worst. Under the Standard harness there are forty-five. When the scaffolding does the heavy lifting of context management, the model’s reasoning capacity stops being the deciding factor.
Third, the Standard harness has an anomaly: low (17.5%) performs worse than none (35.2%). A little reasoning is worse than none at all. ARC Prize doesn’t even mention it, and I have no idea what it means, but I like that they published the row instead of burying it.
What ARC Prize says, which is the part nobody quotes
This is the paragraph that convinced me there was an article here. ARC Prize didn’t publish both numbers as decorative transparency. They published them because the two runs answer two different questions.
“Our Standard harness for ARC-AGI-3 asks how models compare under the same minimal, provider-neutral interface. […] Alternatively, there is a separate question: how well does a model perform when it can use the context-management features its provider designed for it?”
Both questions are legitimate, and the second is arguably more useful if what you want to know is what you’ll actually get in production. But the cross-vendor comparison is something ARC Prize reserves for the first: “the shared interface also gives us a consistent, apples-to-apples comparison across providers.” Going forward they’ll report both conditions on the leaderboard, each clearly labelled.
Now go back to that row in the announcement: Astra’s 99.9% against Opus 5’s 30.2%. The first is Provider Adapter. The second is Standard harness, because there’s nothing else to run. The apples-to-apples comparison is 62.7 against 30.2 — still an excellent result, for the record — but the one occupying the row is 99.9 against 30.2.
In OpenAI’s defence: they said so. In the footnote to that very row: “On ARC-AGI-3, GPT-6 Astra was run with our responses API harness, which changes two settings to better match real-world performance. The changes do not specifically target ARC-AGI-3.” It’s disclosed, it’s honest, and nobody reads it. Which is exactly what a footnote is for.
For the avoidance of doubt: ARC Prize themselves clarify that “while we believe Astra represents meaningful progress towards generalization, we are not claiming that it is AGI.” And François Chollet, who expected ARC-AGI-3 to hold for a year and watched it fall in six months — “about 2x faster than expected,” he wrote — holds the line he’s always held: solving it is not proof of AGI, and it was never meant to be a finish line.
We’re not measuring models any more
This is where the detail stops being trivia about one benchmark.
For a few years we could pretend we were measuring models. One model, one prompt, one answer, one score. That’s over. What gets measured today is the model plus everything we’ve built around it: context management, memory, the tool loop, retry policy, compaction, the agent scaffolding. And that bundle doesn’t fit in a table cell, so it doesn’t get declared.
Take the least suspicious example I can think of, because no lab publishes it. Artificial Analysis’ Coding Agent Index — one of the more serious independent indices out there — puts Claude Fable 5.1 at 70 and GPT-6 Astra at 67. But read the methodology and they tell you themselves that they measure “model+harness (agent framework/scaffold) combinations rather than models alone”: Fable 5.1 running in Claude Code against Astra running in Codex. And they drive it home: “public benchmark rows represent agent variants, not just model names.”
That isn’t a criticism of Artificial Analysis, who document exactly what they do. It’s that there’s no longer a way to do it otherwise. A 2026 frontier model without its scaffolding isn’t an object you can evaluate. It’s a loose part.
And the labs know it. In footnote 16 of the Astra announcement, about its internal computer-use safety benchmark, OpenAI writes: “When we test across third-party models, we use a simpler research setup. Codex has a more complex production configuration, which can result in different raw-model error rates. Provider-side safeguards and computer-tool implementations still differ.” It’s said. It’s in sixteenth position among the footnotes, in an announcement post. But it’s said.
The fine print
Once you put the harness lens on, the rest of the launch numbers start moving too. Here’s what I found, in ascending order of discomfort.
The independent evaluators disagree. Epoch AI ranks Astra first out of 267 models with an ECI of 169, ahead of Claude Fable 5.1 (163) and Claude Opus 5 (162). Artificial Analysis, with its Intelligence Index as it stood on September 3rd, had it tied with its own predecessor at 61 and five points behind Fable 5.1. Both are good evaluators and both publish their methodology; they simply measure different things and weight them differently. Looking at the intervals helps: Astra’s runs 165 to 174, Fable 5.1’s 160 to 166. They overlap. Epoch said as much: the jump “is within our uncertainty range” for the reasoning-era trend.
Saturated benchmarks are doing headline duty. GPQA Diamond at 96% no longer discriminates anything: Gemini 3.8 Flash is at 95.3 and Fable 5.1 at 93.7, all inside the noise. And the 100% on ExploitBench says far more about the benchmark than the model — because on the evaluation OpenAI built from twenty high-severity V8 vulnerabilities across thirteen stable Chrome releases from June to August 2026, that is, with fresh problems, the same model drops to 39%.
Six of the published results are internal. “Internal Design Tasks” 50.0%. “Internal Data Science Tasks” 40.9%. “Internal Database Migration Tasks” 63.9%. “Internal hallucination benchmark” 4.2%. Nobody outside OpenAI can reproduce them, or know whether the bar went up or down relative to the previous version of the same internal benchmark.
The one that most resembles your actual job is missing. GDPval is OpenAI’s own benchmark for economically real tasks, and it appears nowhere in the Astra materials — for a model explicitly sold for professional work. In Artificial Analysis’ independent variant, GDPval-AA v2, Astra regresses by around 80 Elo points against its predecessor. In fairness, the same sentence records an ~80-point gain on AA-Briefcase, so it isn’t a collapse: it’s a shift. But the absence and the regression point the same way.
Whoever funds it sometimes also grades it. OpenAI reports 97.6% on FrontierMath Tier 4. On the benchmark’s own page, Epoch AI writes: “FrontierMath was developed with funding from OpenAI, who has exclusive access to a subset of the benchmark.” That doesn’t make the result false: Epoch ran the obvious check on an earlier model, comparing held-out problems against visible ones, and the result was mixed and statistically insignificant in either direction. So: no sign of overfitting, but no acquittal either. What is certain is that nobody has verified that 97.6% from outside.
And the ecosystem has its own cheating problem now. A June 2026 paper audited 1,968 tasks across five terminal-agent benchmarks and found that 323 — 16% — are hackable by frontier models given only the task description: you can extract the answer without solving the problem. Artificial Analysis took it seriously and shipped v1.4 of its coding agent index in August, which scores reward-hacked attempts as zero and explicitly defines retrieving the answer from outside the environment as reward hacking. But for months there were published tables carrying that contamination inside them.
One last piece of hygiene, because I’ve seen this figure floating around unattached all week: yes, Artificial Analysis measured the “hallucination rate” dropping from 92% to 51% — from GPT-5.6 Sol to Astra, both at max effort. But that metric isn’t what it looks like. It measures the proportion of incorrect answers out of all non-correct responses: how often the model makes something up instead of admitting it doesn’t know. GPT-5.6 Sol was not hallucinating 92% of the time. The improvement is real and it matters; the headline going around does not.
How to read a benchmark table without hurting yourself
This is what I’m taking with me, and what I’ll be asking before I believe a row again:
- Which harness? If it doesn’t say, the score isn’t the model’s. It belongs to a system you only know one part of.
- Same harness for everyone in the comparison? If the model’s owner ran theirs with their own scaffolding and the rivals with the generic one, the table isn’t a comparison. It’s a demo.
- At what effort, and at what price? Almost everything published is run at maximum effort. A single ARC-AGI-3 run under the Standard harness cost $26,098. The human baseline on the same test came in at roughly $12.78 per game attempted, before bonuses. That comparison is part of the result too.
- Is it saturated? Any figure above 95% on a two-year-old benchmark is measuring the benchmark, not the model.
- Who funded it and who has access? Not as an accusation. As context.
- Is it reproducible? If the benchmark is internal, the figure is a marketing claim wearing the clothes of data. It may well be true. It isn’t evidence.
- What’s missing from the table? What a lab quietly stops publishing between versions usually says more than what it publishes.
Verdict
The model, yes. The score, no.
Astra’s progress is real and I have no interest in being a killjoy about it. On FrontierMath Erdős — 68 problems still open as of August 2026, formally verified in Lean, with a budget of $300 and 72 working hours per problem — it solved two: it disproved one by finding a counterexample, for $218 and fifteen hours, and proved another. Every other model evaluated solved zero. That result doesn’t depend on a friendly harness: either the problem gets solved or it doesn’t.
What doesn’t hold up is the naive reading of the table. And that worries me less for this particular launch than for what it signals: we’re entering a phase where a model’s score depends more and more on proprietary infrastructure its maker controls, doesn’t publish, and can change without notice. The benchmark stops measuring a comparable object and starts measuring an integrated product. When that finishes happening, comparing two models will be about as informative as comparing two cars by the lap time each manufacturer posts on its own private track.
The score you’re shown is model × harness × effort. Of those three factors, you’re usually told one.
Ideas over code; evidence over hype.
References
- OpenAI (2026). GPT-6 Astra: A new generation of intelligence, September 3rd, 2026.
- ARC Prize (2026). OpenAI’s GPT-6 Astra on ARC-AGI-3 and full results by effort level and harness.
- Epoch AI. GPT-6 Astra, Announcing FrontierMath Erdős and FrontierMath Tier 4 (v2).
- Artificial Analysis (2026). Benchmarking GPT-6 Astra, Coding Agent Index methodology and the AA-Omniscience definition.
- Zhong, Segal, Bercovich, Saxena, Zhang and Raghunathan (2026). Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops. arXiv:2606.08960, June 8th, 2026.
Leave a Reply