One model. One benchmark. Two scores: 7.8% and 38.3%.
Nothing about the weights changed between those two runs. What changed was the code wrapped around the model — the loop that feeds it observations, keeps its history, and decides what to throw away when the context gets long. That code is the harness, and this week it turned out to be worth about five times the model's score.
If you build agents, this is the most useful thing to happen to benchmarking in months.
The benchmark nobody could beat
ARC-AGI-3 launched on 25 March 2026 from the ARC Prize Foundation — the group co-founded by François Chollet and Mike Knoop. It broke from the previous ARC generations in one important way: it is interactive.
Earlier ARC tasks were static grid puzzles with visible input-output pairs. ARC-AGI-3 drops an agent into a game environment with no instructions, no stated goal, and no described rules. The agent has to work out what the environment is, what counts as progress, and how to get there — the way you would if someone handed you an unfamiliar arcade cabinet and walked away.
The environments are built to a specific spec: fully human-solvable, no pre-loaded knowledge, no hidden prompts, meaningful feedback, and enough novelty that memorisation does not help. The four capabilities being measured are worth writing down, because they are exactly the four things production agents are bad at:
- Exploration — deliberately acting to gather information.
- Modeling — turning raw observations into a world model that generalises.
- Goal-setting — deciding what a good future state looks like without being told.
- Planning and execution — mapping a path, then course-correcting on feedback.
At launch, humans solved every environment. Frontier models scored under 1%. Gemini 3.1 Pro managed 0.37%, GPT-5.4 High 0.26%, Claude Opus 4.6 0.25%, Grok 4.2 a flat zero. That is not "needs work" territory — that is a capability the models did not have.
ARC Prize 2026 put over $2 million behind closing that gap, with the Grand Prize reserved for the first agent to score 100%.
Then the numbers moved
Four months later the leaderboard looks different. Claude Opus 4.8 reached 1.5%. Then Claude Opus 5 posted 30.2% — roughly twenty times Opus 4.8 and the first score that suggests the category is actually tractable.
GPT-5.6 Sol scored 7.8% on the official harness.
And then OpenAI published a post arguing that 7.8% was not a measurement of GPT-5.6 Sol at all.
The 7.8% / 38.3% problem
OpenAI's claim is specific and, unusually for a benchmark dispute, mechanical rather than rhetorical. The official ARC-AGI-3 harness did two things that happen to be catastrophic for the way GPT-5.6 Sol was built:
It discarded reasoning after every move. The harness kept the record of what the model did, but dropped the private reasoning explaining why. The model could see "I moved the purple block" without retaining the hypothesis that made moving it worth trying.
It used rolling truncation. Once the conversation passed roughly 175,000 characters, the oldest content was dropped outright. On a benchmark whose entire point is accumulating knowledge about an unfamiliar environment over many turns, deleting the earliest observations removes precisely the observations that took the most effort to acquire.
The result, in OpenAI's description, was a model that "kept dwelling on individual actions and failed to build on what it had already learned."
OpenAI reimplemented the harness with two settings that already ship in the Responses API — retained reasoning and compaction — and re-ran the public task set. The score went from 13.3% to 38.3%, using roughly six times fewer output tokens per game. On one environment the model cleared all six levels, where no frontier model on the public leaderboard had cleared level one.
Fewer tokens and five times the score. That is the shape of a fix, not a workaround.
What the two settings actually do
Retained reasoning
Reasoning models produce a private chain of thought that is not returned to you. In a stateless chat-completions loop, that reasoning evaporates the moment the response ends — the next turn starts from the visible transcript only.
The Responses API keeps it. Pass the previous response ID and the model's own prior reasoning is carried into the next turn, encrypted and hidden from the client but available to the model.
The difference between these two loops is not stylistic:
# Stateless: reasoning is discarded every turn
history = []
for step in range(max_steps):
history.append({"role": "user", "content": observe(env)})
response = client.responses.create(
model="gpt-5.6-sol",
input=history[-WINDOW:], # rolling truncation drops the oldest turns
)
act(env, response.output_text)# Stateful: reasoning persists across turns
previous_id = None
for step in range(max_steps):
response = client.responses.create(
model="gpt-5.6-sol",
input=[{"role": "user", "content": observe(env)}],
previous_response_id=previous_id, # prior reasoning carries forward
store=True,
)
previous_id = response.id
act(env, response.output_text)Note the second loop passes only the new observation each turn. The chain is maintained server-side.
Compaction
Truncation deletes. Compaction summarises. When context approaches its limit, the model produces a compaction item — an encrypted, token-efficient representation of prior state and reasoning that replaces the raw history rather than dropping it.
The key detail is that recent models are trained to produce these items. It is not a summarisation prompt bolted on top; it is a native operation aligned with how the model represents state internally.
# Illustrative — check the current Responses API reference for exact signatures
if response.usage.total_tokens > COMPACT_THRESHOLD:
compacted = client.responses.compact(response_id=response.id)
previous_id = compacted.idThis is the same lesson we covered in the 1M context myth: the harness cap, not the advertised window, is what your agent lives inside — and how the harness enforces that cap determines whether long sessions accumulate knowledge or leak it.
The line between a setting and a cheat
The obvious objection: if every lab brings its own harness, the leaderboard measures engineering budgets rather than models.
ARC Prize's position is that primary leaderboard rows should be model plus reasoning settings, not full Codex- or Claude-Code-style harnesses, and that comparisons need one methodology across providers. Chollet drew the line more precisely: harnesses "custom-made to solve the benchmark" are prohibited, while "general-purpose API settings available to all API users" are permissible — with settings and cost reported transparently.
That line is defensible, and the evidence for why it matters is uncomfortable. In ARC Prize's own testing, a harness engineered against specific public environments pushed Opus 4.6 to 97.1% on one environment and 0% on another. Harness gains that come from task-specific tuning do not generalise at all. Gains that come from not throwing away the model's memory presumably do.
Retained reasoning and compaction are documented, generally available features that any developer can switch on. They are closer to "we were running the model wrong" than to "we tuned for the test." But the incident still leaves a real problem: a benchmark that intends to measure models is, in practice, measuring model-plus-harness, and there is no neutral harness. Every choice — truncation policy, reasoning persistence, tool schema, retry logic — is a thumb on the scale for someone.
Hacker News split predictably into "breakthrough" and "benchmaxxing." Both camps are describing the same fact from different sides.
What to take into your own agents
Whatever you conclude about the leaderboard, the engineering lesson transfers directly:
- Never let a rolling window silently delete state. If your agent loop truncates on a character or token threshold, you are deleting your agent's hardest-won knowledge first. Summarise or compact instead. This is the single highest-leverage change in most agent loops.
- Persist reasoning across turns if your provider supports it. An agent that forgets why it did something re-derives it, badly, every turn. That is the "dwelling on individual actions" failure mode — and it costs tokens while producing worse output.
- Measure your harness, not just your model. Before switching models because results are poor, re-run the same task with reasoning persistence and compaction enabled. A 5× swing is available for free in some loops. Our notes on harness engineering for production agents go deeper on how to structure this.
- Treat published benchmark numbers as harness-conditional. A score is a claim about a configuration. When a vendor quotes one, the useful follow-up is what loop produced it — the same skepticism we applied to broken coding benchmarks.
- Longer context is not the fix. Compaction beat truncation while using six times fewer tokens. The win came from keeping the right things, not from keeping more. That is context engineering, not capacity shopping.
The uncomfortable conclusion
ARC-AGI-3 was designed to test whether an agent can explore an unfamiliar world, build a model of it, and act on that model over a long horizon. It turns out a large share of what looked like a missing capability was a missing memory — thrown away by the plumbing between the model and the environment.
That should make you slightly more optimistic about frontier models and considerably less confident about your own agent loop. Most production agents in the field today truncate rather than compact, and start each turn from a transcript with the reasoning stripped out.
If you have been blaming the model, check the harness first. It is cheaper to fix, and this month it was worth 30 points.
Sources: ARC Prize — ARC-AGI-3, OpenAI — How two settings tripled our ARC-AGI-3 scores, The Decoder, OpenAI Compaction guide