Part Ⅵ · The compositional benchmark

ARC-AGI-2 the gap reopens — and cost goes on the board.

Built to undo o3. It keeps the grid format but curates tasks that compose multiple interacting rules and define symbols in context — the fluid novelty that defeated 2024-era frontier models — and, for the first time, it scores cost per task. About 1,455 teams entered the first season; the top private score was 24%, and the 85% grand prize went unclaimed.

100%

Humans solve

24%

Top contest score

54%

Highest (costly)

85%

Grand prize unclaimed

An official ARC-AGI-2 contextual-rule task with a solved demonstration above and an unsolved test grid below — See the benchmark first

Same canvas. Several rules at once.

Read the demonstrations as a tiny invented language: identify the objects, infer what each colour or arrangement means *in this task*, and work through the transformations in the right order.

**Decompose the scene.** Find objects, boundaries, markers, and relations.

**Compose the rules.** Determine order and context.

**Build and verify.** Replay your rule against every demonstration.

Try this task ↗ Official benchmark ↗

Official contextual-rule task image and browser puzzle: ARC Prize Foundation.

What it is

Same grids. Harder rules. A price tag.

ARC-AGI-2 looks like ARC-AGI-1 — coloured input→output grids, a rule to infer — but the tasks are curated to need several interacting rules, sequential steps, and symbols defined in the demonstrations themselves. Every task is solved by at least two humans; in the launch-era evaluation, directly prompted frontier models fell to the low single digits.

The second change is on the scoreboard: cost-per-task is now a first-class axis. A cheap system that scores a little less can beat an expensive one that scores a little more. This is the benchmark's direct answer to o3, which cleared ARC-AGI-1 at thousands of dollars a task.

The result was a benchmark that did its job. The gap ARC-AGI-1 had nearly closed reopened to a chasm — and stayed open through the whole first season.

The direct literature

Papers that define ARC-AGI-2.

Start with the benchmark definition and the 2025 season reportRoughly 1,455 teams filed about 15,000 entries, and the paper track received about 90 submissions; the top private score still reached only 24%.. The adjacent method papers show what the cost axis rewards: efficient adaptation and compression rather than unconstrained search.

Category 1

Reference papers

The benchmark definition and the official account of the first ARC-AGI-2 season.

2 papers

№02 paper 2025

ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems

Chollet et al. · arXiv · 2025

Benchmark

After o3 cracked ARC-AGI-1, a harder compositional redesign that puts cost on the leaderboard so brute-force compute stops looking like intelligence

Its job in the ARC-AGI-2 storyThe benchmark definition. ARC-AGI-2 keeps the grid format but curates tasks that need multiple interacting rules, sequential steps, and in-context symbol definition — exactly what frontier reasoning models fail at — and promotes cost-per-task to a first-class axis. It is the document that reopened the human–machine gap after o3.

~3%ARC-AGI-2 (o3, semi-private)

>=2 solvers/taskHuman calibration

~24%Best 2025 entry (offline)

85% (unclaimed)2025 grand-prize bar

Read the original ↗Full entry in Part Ⅱ →

№05 report 2026

ARC Prize 2025: Technical Report

Chollet et al. · arXiv · 2026

BenchmarkSynthesis

The first ARC-AGI-2 competition: top score just 24%, and the year's winning pattern is a per-task propose-verify-iterate refinement loop

Its job in the ARC-AGI-2 storyThe ARC Prize 2025 technical report: the account of the first season run against ARC-AGI-2. It names the year's theme the refinement loop, records the top private score at just 24% with the 85% grand prize unclaimed, and warns about knowledge-dependent overfitting as datasets age.

~24% @ ~$0.20/taskARC-AGI-2 top (private)

~45% AGI-1 / ~8% AGI-2TRM (7M params)

~31% → ~54% @ $30/taskGemini-3-Pro harness

~1,455 teams / ~15k entries2025 competition

Read the original ↗Full entry in Part Ⅱ →

Category 2

Direction papers

Efficient, from-scratch methods that embody what a cost-scored benchmark is meant to reward.

2 papers

№16 paper 2025

ARC-AGI Without Pretraining (CompressARC)

Liao et al. · arXiv · 2025

Neural

A 76K-parameter net with no pretraining or search compresses a single ARC puzzle at inference time — and solves ~20% of ARC-AGI-1

Its job in the ARC-AGI-2 storyThe clearest embodiment of what ARC-AGI-2's cost axis rewards: solve a task by minimizing its description length, with no pretraining and no search over an external program bank. CompressARC's ~20–34% from pure compression is a principled, cheap outsider to the search-and-adapt mainstream — and a live claim that MDL may be the objective the other families only approximate.

~20%ARC-AGI-1 eval (pass@2)

34.75%ARC-AGI-1 train (pass@2)

76KParameters

~20 min (RTX 4070)Compute / puzzle

Read the original ↗Full entry in Part Ⅱ →

№13 paper 2024

Searching Latent Program Spaces (LPN)

Bonnet et al. · arXiv 2024 · NeurIPS 2025 spotlight

NeuralTest-time

Learn a continuous latent space of programs and search it with gradients at test time — no DSL, no per-task fine-tuning

Its job in the ARC-AGI-2 storyA cheaper, differentiable form of per-task adaptation: search a learned latent space of programs by gradient descent instead of fine-tuning weights or sampling thousands of candidates. It points at the efficient adaptation ARC-AGI-2's cost scoring is meant to reward.

78.5% (vs 57% TTT)ARC-AGI in-distribution

15.5% (vs 15.25% TTT)ARC-AGI OOD

~2× (7.75→15.5%)OOD gain from test-time search

178M params, 256-dim latentModel

Read the original ↗Full entry in Part Ⅱ →

What scored, and how

The best of the first season, best first.

Ranked by raw score — but read the cost column, because that is the point. The highest scorerThat was a Gemini-3-Pro–class harness: wrapping the model in a verifier that fed failures back climbed it from ~31% to ~54% on ARC-AGI-2 — at about $30 per task. paid ~150× the contest winner's cost per task; on ARC-AGI-2's scored axis, the cheap system won.

Gemini-3-Pro harnessapplication-layer refinementRefinement loop 31% → 54% @ $30/task

Wrap a frontier model's chain-of-thought in a verifier that runs each candidate program against the demonstration pairs and feeds failures back as the next prompt, iterating until a candidate replays every pair.

Why it worksThe highest raw score of the 2025 season, and the clearest demonstration that the refinement loop — propose, verify, feed back, iterate — lifts a general model well past its single-pass ceiling on compositional tasks.

Where it stopsIt gets there at roughly $30 per task — about 150× the contest winner's cost — so on ARC-AGI-2's cost-scored axis it is a strong result, not an efficient one.

2025 Kaggle winnertop private entrySynthetic data + TTT ~24% @ ~$0.20/task

Generate millions of synthetic look-alike tasks, then test-time-train a compact (~4B-parameter) model on the task at hand — reaching the top private score at about twenty cents per task.

Why it worksThe top score of the first ARC-AGI-2 competition, and the point of the cost axis made concrete: cheap and small beat expensive and large, at roughly 150× less cost than the highest-scoring harness.

Where it stopsTwenty-four percent is still far below the 85% grand-prize bar and further below humans' ~100% — synthetic-data TTT narrows the gap but does not close it on genuinely compositional novelty.

Tiny Recursive Model (TRM)weight-space refinementZero-pretraining net ~45% AGI-1 / ~8% AGI-2

A ~7-million-parameter network with no pretraining fits its weights to each task using architectural recursion in place of search depth — astonishing on ARC-AGI-1's simpler tasks.

Why it worksProof that a tiny, from-scratch model can reach the mid-40s on ARC-AGI-1 without any pretraining — a striking efficiency result and a direct challenge to the scale narrative.

Where it stopsIt drops to roughly 8% on ARC-AGI-2's compositional tasks — the honest measure of how much harder interacting-rule tasks are than single-transformation ones.

Frontier reasoning modelso3 · o1-pro · Claude 3.7Frontier reasoning ~1–3% harness-free

The strongest general models prompted directly, with no ARC-specific scaffolding — fitting reasoning chains end-to-end and collapsing on genuinely novel composition.

Why it worksAn honest harness-free launch baseline, and evidence that ARC-AGI-2 did its job: models that cleared 87.5% on ARC-AGI-1 fell to the low single digits here.

Where it stopsSingle-digit accuracy against ~100% human is the reopened gap. Compositional, in-context-defined tasks are exactly the fluid generalization these models do not have.

What the winner talks reveal about the pipeline

The interviews make the shared machine more concrete. NVARC pushed expensive synthetic-puzzle generation offline, distilled that curriculum into compact models, then used task-local fine-tuning and a diverse TRM ensemble component at evaluation time. the ARChitects replaced their 2024 text-like autoregressive grid system with 2D-aware masked diffusion, recursive repair, and perspective scoring. MindsAI combined fine-tuning, augmentation ensembles, tokenizer dropout, and pretraining changes. The paper winners complete the loop from three directions: TRM recursively refines latent and answer states, SOAR learns from its own program-search traces, and CompressARC uses minimum description length to learn from one puzzle. The common unit is not a model alone; it is a representation, a proposal engine, feedback, and an adaptation protocol. Read the video notes and evidence basis in Appendix H →

The cost axis

Cheap and small beat costly and large.

ARC-AGI-2's headline change is that score is read against priceDollar cost is a proxy, not the quantity itself — it conflates model pricing, hardware, and provider margins. The open question the paper flags is what a vendor-neutral efficiency unit should be: compute, or actions counted against a human baseline.. The ~$0.20 test-time-trained model won the contest at 24%; the ~$30 harness scored higher but off the efficiency frontier. Plot score against cost and the tradeoff is impossible to miss.

Figure · Score vs cost

Selected ARC-AGI-2 score–cost frontiers

Higher and cheaper is better. Frontier status is computed within each evaluation class, among the systems shown.

Question answered: Which selected ARC-AGI-2 systems buy more score for their reported inference cost?

Frontier status is computed only within the same declared evaluation class and only among the systems shown. Contest, commercial-model, and refinement results share axes for orientation but are not treated as one controlled evaluation.

2025 contest · private evalARC Prize verified · commercial modelsARC Prize verified · refinement harness Non-dominated within shown evaluation class

Underlying score, cost, protocol, and frontier status.
System	Score	Cost/task	Protocol	Status
NVARC	24.03%	$0.2	2025 Kaggle private evaluation; contest constraints	Frontier among shown peers
Gemini 3 Pro	31%	$0.81	ARC Prize verified commercial baseline	Frontier among shown peers
Opus 4.5	37.6%	$2.2	Thinking, 64k; ARC Prize verified	Frontier among shown peers
Poetiq + Gemini 3 Pro	54%	$30	Open refinement harness; ARC Prize verified	Frontier among shown peers

Snapshot 2025-12-05. Source: ARC Prize 2025 Results & Analysis.

The solution space

Everything that was tried — and the target it left open.

Six families, one gap. The refinement loop and synthetic-data adaptation dominate; tiny from-scratch netsThe sharpest case is the Tiny Recursive Model: ~7M parameters, no pretraining, ~45% on ARC-AGI-1 — then ~8% on ARC-AGI-2. One net, two benchmarks: the drop locates the difficulty in the tasks, not the parameter count. and pure frontier reasoning bound the space. What none reach is the benchmark's actual target: compositional generalization from few examples, cheaply.

Dominant

Synthetic data + test-time training

Generate millions of synthetic tasks that resemble ARC, pretrain or augment on them, then test-time-train a compact model on the actual task. Small models plus cheap per-task adaptation, tuned to win on the cost axis rather than the raw-score axis.

e.g. The 2025 ARC-AGI-2 Kaggle winner: a ~4B-parameter model at ~24% for about $0.20/task

Dominant

Refinement loop over a frontier model

The defining move of 2025: wrap a commercial model in a verifier that runs each candidate against the demonstration pairs and feeds the error back, iterating toward a program that replays them. It lifts scores the most — and costs the most.

e.g. The Gemini-3-Pro application-layer harness: ~31% → ~54% at about $30/task

Tried

Evolutionary / program synthesis with a verifier

The same propose-verify-iterate idea in a more explicit search: breed candidate programs, score them against the demos, and recombine the fittest — increasingly over natural-language solution descriptions rather than raw code, for a less brittle search space.

e.g. Berman's 2025 evolutionary direction; the general refinement-loop family named in the 2025 report

Emerging

Zero-pretraining recursive networks

Fit a tiny network to each task from scratch, using architectural recursion in place of search depth or pretraining. A pure efficiency play: minimal parameters, no external knowledge, adaptation baked into the architecture.

e.g. The Tiny Recursive Model (~7M params): ~45% on ARC-AGI-1, ~8% on ARC-AGI-2

Tried

Pure frontier reasoning (harness-free)

The strongest general models, prompted directly with no scaffolding — the official harness-free baseline. On ARC-AGI-2 this is the floor: end-to-end reasoning chains collapse on tasks that compose multiple interacting rules.

e.g. o3, o1-pro, Claude 3.7 harness-free: low single digits vs ~100% human

Open gap

Compositional generalization at low cost

The actual target the benchmark defines and nothing yet reaches: solve tasks that compose several interacting rules and define symbols in context, cheaply and from few examples. The 85% grand prize sits here, unclaimed, and the top result is 24%.

e.g. No system near the 85% grand-prize bar; the field's ceiling is ~24% private

The tradeoffs that shape it

Accuracy vs cost per task

cheap: ~$0.20/task (synthetic-data TTT) costly: ~$30/task (frontier-model harness)

ARC-AGI-2's headline change: cost is scored. The cheap ~4B TTT model won the contest at 24% while the harness scored higher (~54%) at ~150× the price — so which system is 'best' depends on which axis you read.

Big frontier model vs small from-scratch net

a commercial frontier model in a harness a ~7M-parameter network, no pretraining

Tiny, from-scratch models reach ~45% on ARC-AGI-1 but fall to ~8% on ARC-AGI-2, suggesting that compositional difficulty matters more than parameter count alone.

Single-pass vs refinement loop

one forward pass, prompted directly propose → verify → feed back → iterate

In the 2025 evaluation, harness-free frontier models occupied the single-digit floor; wrapping the same model in a verify-and-retry loop lifted it into the tens of percent. The loop was the lever — the demonstration pairs supplied the feedback that made it converge.

The state of play

ARC-AGI-2 keeps the grid format but adds multi-rule composition, sequential steps, symbols defined in context, and cost per task. Every task was solved by at least two humans, while general models prompted directly scored in the low single digits during the 2025 evaluation.

The 2025 season centered on a refinement loop: propose, verify against the demonstrations, feed failures back, and iterate. A commercial-model harness reached ~54% at ~$30 per task; compact synthetic-data test-time training won the contest at ~24% and ~$0.20 per task; a tiny from-scratch network reached ~8%. Because cost is scored, the cheaper result tells a different and equally important story.

And yet the benchmark is wide open. The top private score is 24%, the 85% grand prize went unclaimed, and humans sit near 100%. The gap that matters is no longer only accuracy or only cost but both at once: compositional generalization from few examples, cheaply. That two-axis target is exactly what ARC-AGI-3 pushes further — from static compositional puzzles to interactive worlds with no demonstration pairs and no stated goal at all.

1

Is cost-per-task the right second axis, and how should it trade against accuracy?

ARC-AGI-2 made cost a scored dimension, and immediately the cheap ~$0.20 contest winner and the costly ~$30 high-scorer told different stories about who is 'winning.' The open question is how to weigh accuracy against efficiency when declaring progress — and whether an efficiency-normalized metric (accuracy per dollar) is the honest way to read the board.
2

Why do frontier reasoning models collapse on compositional novelty?

In the launch evaluation, models that cleared 87.5% on ARC-AGI-1 fell to the low single digits on ARC-AGI-2. Did end-to-end reasoning chains fail to represent several interacting rules at once? Did in-context symbol definition break their priors? Or were the tasks simply out of distribution? The answer determines whether scale is enough or a different mechanism is needed.
3

How do you refine against the demos without overfitting to them?

The refinement loop verifies candidates against the demonstration pairs, but compositional tasks make the demo→test generalization gap sharper: a program can replay every demo and still miss the held-out input. Is there a principled regularizer against demo-overfitting, or is candidate diversity the only defense?
4

How do you keep the benchmark honest as data leaks into training corpora?

The 2025 report warns of knowledge-dependent overfitting as ARC-style tasks age into training data. Are private, rotating test sets enough to prevent contamination, and how should human baselines be measured and re-measured so the comparison stays fair as both sides move?

ARC-AGI-2 turned the gap into a two-axis problem: compositional generalization, cheaply. That is the target still open at 24% — and the direct setup for the interactive frontier.

→ Next: Part Ⅶ · ARC-AGI-3, where the demonstrations disappear

ARC-AGI-2 the gap reopens — and cost goes on the board.

Same canvas. Several rules at once.

Same grids. Harder rules. A price tag.

Papers that define ARC-AGI-2.

Reference papers

ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems

ARC Prize 2025: Technical Report

Direction papers

ARC-AGI Without Pretraining (CompressARC)

Searching Latent Program Spaces (LPN)

The best of the first season, best first.

Cheap and small beat costly and large.

Selected ARC-AGI-2 score–cost frontiers

Everything that was tried — and the target it left open.

Synthetic data + test-time training

Refinement loop over a frontier model

Evolutionary / program synthesis with a verifier

Zero-pretraining recursive networks

Pure frontier reasoning (harness-free)

Compositional generalization at low cost

The tradeoffs that shape it

The state of play

Is cost-per-task the right second axis, and how should it trade against accuracy?

Why do frontier reasoning models collapse on compositional novelty?

How do you refine against the demos without overfitting to them?

How do you keep the benchmark honest as data leaks into training corpora?

ARC Radio

Help improve this guide

Review received