Part Ⅴ · The original benchmark

ARC-AGI-1 five years unbeaten, then a step-function.

The 2019 benchmark that started it all. A handful of input→output grids, a hidden rule to infer, and an exact-match verdict. It crept from ~0% to ~5% for three years, until 2024's test-time adaptation drove the private record to 55.5% and OpenAI's o3 vaulted to 87.5% — near-human, at thousands of dollars a task. On accuracy it is effectively solved; on efficiency it is not.

98%

Humans solve

87.5%

Best (o3, high-compute)

2020

First Kaggle win

5 years

Benchmark unbeaten

An official ARC-AGI-1 task: two example grid transformations followed by a test input and an empty output grid — See the benchmark first

Find the rule. Draw the missing grid.

This is not a reflex game. Read each demonstrated grid from left to right, ask what changed — objects, colours, positions, counts, or relations — then test one candidate rule against *every* example.

**Study the demonstrations.** Each input is paired with its correct output.

**Infer the transformation.** One rule must explain every pair.

**Draw the test output.** Only an exact grid counts.

Play this task ↗ Official benchmark ↗

Official task image and browser puzzle: ARC Prize Foundation.

What it is

A few examples. One rule to find.

Each ARC-AGI-1 task shows a handful of coloured input→output grid pairs and one test input. The solver must infer the single transformation the pairs share and produce the exact output grid — no partial credit, no natural-language hints.

Every task is novel and few-shot, and is designed to rely on Core Knowledge priors such as objectness, geometry, counting, and agentness. This makes memorization and scale less useful. The goal is to separate skill from intelligence: not how much a system knows, but how efficiently it learns a new rule.

The benchmark held that line for a remarkably long time. A hand-built symbolic searcher won in 2020 and stayed on top for three years while large language models stalled below 5% — until per-task adaptation, and then frontier reasoning, finally moved it.

The direct literature

Papers that define ARC-AGI-1.

Start with the benchmark definition and the 2024 season reportThe ARC Prize 2024 Technical Report records the private score's rise from ~33% at the end of 2023 to 55.5%, and credits per-task adaptation rather than scale alone.. Then read the small set of solution papers that moved the record. Full catalog entries live in Part Ⅱ; the next section ranks the major results.

Category 1

Reference papers

The definition of the benchmark and the official account of the season that broke it.

2 papers

№01 paper 2019

On the Measure of Intelligence

François Chollet · arXiv · 2019

BenchmarkFoundations

Chollet redefines intelligence as skill-acquisition efficiency and builds ARC: few-shot grid puzzles easy for humans, hard for machines

Its job in the ARC-AGI-1 storyThe founding document. It defines intelligence as skill-acquisition efficiency, argues that a fair test must control for prior knowledge and experience, and introduces ARC as the concrete benchmark built on innate Core Knowledge priors — the yardstick every ARC-AGI-1 solution is measured against.

~1,000ARC tasks

~3-5Demos per task

30×30Max grid

Read the original ↗Full entry in Part Ⅱ →

№04 report 2024

ARC Prize 2024: Technical Report

Chollet et al. · arXiv · 2024

BenchmarkTest-time

The 2024 post-mortem: test-time adaptation, not scaling, drove ARC-AGI-1's private SoTA from ~33% to 55.5% in one season

Its job in the ARC-AGI-1 storyThe ARC Prize 2024 technical report: the official account of the season that broke the benchmark open. It names test-time adaptation as the lever, records the climb from a stalled ~33% to 55.5% private, and frames o3's 87.5% as the result that forced ARC-AGI-2.

33% → 55.5%ARC-AGI-1 SoTA (private)

53.5%Kaggle winner (ARChitects)

~11%No-adaptation ceiling

~42%Greenblatt GPT-4o synthesis

Read the original ↗Full entry in Part Ⅱ →

Category 2

Solution papers

The works that each introduced a lever the record turned on — search, sampling, adaptation, reasoning.

6 papers

№07 code 2020

ARC 2020 Kaggle 1st-Place Solution (Icecuber)

Johan S. Wind · Kaggle ARC · 2020

MilestoneFoundations

Solo Kaggle winner: 142 hand-built grid ops brute-force-composed into a DAG of pieces — ~20% that defined ARC-AGI-1 solving for three years

Its job in the ARC-AGI-1 storyThe 2020 Kaggle winner established the early template: compose a hand-built DSL of ~142 grid operations and keep programs that pass an exact-match verifier. Its symbolic search led for three years while large models stayed below 5%.

~20%ARC-AGI-1 private (1st)

129/419Public eval, depth 2

~142 fnsDSL size

~9hDepth-3 runtime

Read the original ↗Full entry in Part Ⅱ →

№10 blog 2024

Getting 50% (SoTA) on ARC-AGI with GPT-4o

Greenblatt et al. · Redwood Research blog · 2024

SynthesisTest-time

Sample ~8,000 GPT-4o Python programs per task, keep only those that replay every training pair, majority-vote the survivors — 50% on ARC-AGI

Its job in the ARC-AGI-1 storyThe proof that sample count is a real axis of accuracy: draw thousands of GPT-4o Python programs per task, keep only those that replay every demonstration pair, then vote and repair. The direct ancestor of the 2025 refinement loop.

~50%ARC-AGI public eval (100-task)

~42%ARC-AGI-Pub semi-private

~8,000Programs sampled / task

+13 ptsRevision gain

Read the original ↗Full entry in Part Ⅱ →

№08 paper 2024

The Surprising Effectiveness of Test-Time Training for Abstract Reasoning

Akyürek et al. · arXiv · 2024 · ICML 2025

Test-timeSynthesis

Briefly rewrite an 8B model's own weights on each ARC task's demos and it reaches ~average-human accuracy — a 6x leap over frozen fine-tuning

Its job in the ARC-AGI-1 storyThe paper that formalized test-time training for ARC: fit a fresh LoRA adapter to each task's own demonstrations, a ~6× gain that reaches average-human level when ensembled with program synthesis. The mechanism that defined the 2024 season.

53.0%ARC-AGI-1 pub val (8B, TTT)

61.9%+ program-synthesis ensemble

up to 6xvs fine-tuned baseline

57.8% (+7.3)BIG-Bench Hard (10-shot)

Read the original ↗Full entry in Part Ⅱ →

№09 paper 2025

Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective

Franzen et al. · arXiv · ICML 2025

Test-timeMilestone

The 2024 ARC Prize Kaggle winner: score each candidate under every symmetry of the task, multiply, and keep only the jointly-plausible answers

Its job in the ARC-AGI-1 storyThe 2024 Kaggle winner: test-time training on an open 8B model, then score every candidate under all task symmetries and multiply the probabilities so only jointly-plausible answers survive — all offline at about two cents per task.

53.5%ARC Prize 2024 (private)

71.6%ARC-AGI-1 public eval

~$0.02Cost/task

~73%ConceptARC (OOD)

Read the original ↗Full entry in Part Ⅱ →

№11 blog 2024

How I got a record 53.6% on ARC-AGI-Pub using Sonnet 3.5 with Evolutionary Test-time Compute

Jeremy Berman · Substack · 2024

SynthesisMilestone

A genetic algorithm over Sonnet-written Python transforms — breed, score, recombine — reaches a record 53.6% on ARC-AGI-Pub, the mark before o3

Its job in the ARC-AGI-1 storyThe open, reproducible record on ARC-AGI-Pub before o3: a genetic algorithm that breeds Claude-written Python transforms, scores them against the demos, and recombines the fittest across generations — evolutionary program search made to work.

53.6%ARC-AGI-Pub (record)

268/500Challenges solved

~75% vs ~70%Deep vs shallow (60 tasks)

~$10,000API budget

Read the original ↗Full entry in Part Ⅱ →

№06 blog 2024

OpenAI o3 Breakthrough High Score on ARC-AGI-Pub

ARC Prize · ARC Prize blog · 2024

MilestoneTest-time

o3 vaults ARC-AGI-1 from ~5% to 75.7-87.5%, near-human at last — but at tens to thousands of dollars per task

Its job in the ARC-AGI-1 storyThe step-function. At test time o3 searches over many candidate chains-of-thought with a learned evaluator, clearing 87.5% — the first human-competitive score on ARC-AGI-1. Not a competition entry but a frontier-lab result, and the one that triggered ARC-AGI-2.

75.7%ARC-AGI-1 (high-eff)

87.5%ARC-AGI-1 (172× compute)

~$20–thousandsCost/task

<30%Projected ARC-AGI-2

Read the original ↗Full entry in Part Ⅱ →

What won, and how far

Every major solution, best first.

These scores come from different splitsIcecuber's ~20% is from the 2020 private leaderboard; o3's 75.7% and 87.5% are from the 100-task semi-private set; Greenblatt's ~42% is from the public leaderboard. — public evaluation, ARC-AGI-Pub, and private — so read the trajectory, not a single controlled ladder. Symbolic search led for three years, test-time adaptation broke through in 2024, and o3 crossed the human bar at high cost.

OpenAI o3OpenAIFrontier reasoning 75.7% → 87.5%

At test time, search over many candidate chains-of-thought with a learned evaluator and enormous inference compute — the low-efficiency run spent roughly 172× the compute to reach the high score.

Why it worksThe first human-competitive score on ARC-AGI-1, and a demonstration that a general reasoning model, given enough guided test-time search, can clear tasks that stalled every prior method below the mid-50s.

Where it stopsIt costs tens to thousands of dollars per task. Whether it acquired the sample-efficient generalization ARC measures, or expensively simulated it with search, is exactly the 'o3 is not AGI' question — and it forced the harder, cost-scored ARC-AGI-2.

The ARChitectsFranzen, Disselhoff & HartmannTest-time training 53.5% private · 71.6% public

Test-time-train an open 8B model on each task, then apply a product of experts: score each candidate answer under every symmetry of the task and multiply the probabilities, so only answers plausible under all views survive.

Why it worksWon the 2024 ARC Prize Kaggle competition entirely offline at about two cents per task — the strongest fully open, reproducible, cost-disciplined result of the season.

Where it stopsEven a symmetry-hardened product of experts tops out in the low 70s on the public split; the compositional novelty ARC-AGI-2 would introduce is out of reach.

MIT — Test-Time TrainingAkyürek et al. · MITTest-time training 53% public · 61.9% ensembled

Fit a fresh LoRA adapter to each task's own demonstration pairs, with augmentation — a roughly 6× gain over the frozen model, matching average human performance when ensembled with a program-synthesis solver.

Why it worksFormalized and isolated test-time training as the dominant lever of the 2024 season, showing that per-task adaptation, not raw scale, was what moved the record.

Where it stopsThe adapter overfits the handful of demonstrations; the gains ride on ensembling with symbolic search, and there is still no satisfying theory of why weight-space adaptation beats in-context learning on the same examples.

MindsAICole, Osman & HodelTest-time training 55.5% · season high

A transductive deep-learning pipeline with full-parameter test-time fine-tuning and an augment–infer–vote loop — the lineage that pioneered test-time adaptation on ARC.

Why it worksThe highest score of the 2024 season, and the group that first proved test-time fine-tuning could carry ARC from single digits into the mid-50s.

Where it stopsKept private and so ineligible for prize money — a reminder that a headline number is only progress once it is open and reproducible.

Berman — EvolutionaryJeremy BermanLLM program synthesis 53.6% · record before o3

A genetic algorithm over Claude-written Python transforms: generate a population, score each against the demonstration pairs, keep the fittest, and recombine across generations toward a program that replays every pair.

Why it worksHeld the open, reproducible ARC-AGI-Pub record until o3 landed the same month, and pointed at the more abstract 2025 direction of evolving natural-language solution descriptions rather than raw code.

Where it stopsEvolutionary search is brittle and sample-hungry; it verifies against the demos and can still miss the held-out test input when the demos underdetermine the rule.

Greenblatt (GPT-4o)Ryan Greenblatt · RedwoodLLM program synthesis ~42% AGI-Pub · 50% public

Sample about 8,000 GPT-4o Python programs per task, keep only those that reproduce every demonstration pair, majority-vote the survivors, then repair the near-misses.

Why it worksReached 50% on the public split with an off-the-shelf model and no fine-tuning, proving that sheer sample count against an exact-match verifier is a real, tunable axis of accuracy.

Where it stopsThousands of samples per task is expensive and wasteful, and the approach leans entirely on an execution oracle — the demonstration pairs — that ARC-AGI-3 later removes.

IcecuberJohan S. Wind, soloSymbolic · DSL search ~20% · 1st, 2020 Kaggle

A hand-built DSL of about 142 grid functions, brute-force-composed into a DAG of intermediate pieces, keeping only programs that reproduce every training pair.

Why it worksA solo entry with no learning at all won the first public competition and set the DSL + brute-force + exact-match-verifier template that held the frontier for three years.

Where it stopsIts human-authored primitives encode substantial prior knowledge, and the search cannot generalize beyond that grammar. Large models did not beat it until test-time adaptation arrived in 2024.

Table · ARC-AGI-1 selected results

The score ladder, one row per team

Best reported score per team across public-eval, ARC-AGI-Pub, and private splits.

OpenAI o3 · OpenAI

87.5%

MindsAI · Cole, Osman, Hodel

55.5%

Berman — Evolutionary · Jeremy Berman

53.6%

The ARChitects · Franzen, Disselhoff, Hartmann

53.5%

MIT — Test-Time Training · Akyürek et al. · MIT

53%

Greenblatt (GPT-4o) · Ryan Greenblatt · Redwood

42%

Icecuber · Johan S. Wind, solo

20%

The solution space

Everything that was tried.

Five families carried ARC-AGI-1, and one gap remains. The through-line: adapt to the task in front of you, and keep only what an exact-match verifierA clean correctness gate — but a leaky oracle. Several distinct programs can replay every demonstration pair, so it confirms consistency, not intent; when the demos underdetermine the rule, a program that passes them can still miss the held-out test input. confirms. What no family delivered was that accuracy at anything near a human's cost.

Tried

Hand-built DSL + brute-force search

Author a domain-specific language of grid operations (recolor, translate, tile, flood-fill, count) and search compositions of them, keeping only programs that reproduce every demonstration pair under an exact-match verifier. No learning: correctness comes entirely from the verifier and the expressiveness of the grammar.

e.g. Icecuber's ~142-primitive DSL, 1st in the 2020 Kaggle competition at ~20%

Tried

LLM program synthesis (sample-and-verify)

Prompt a language model to write candidate programs (Python or a DSL), run each against the demonstration pairs, and keep only those that replay them exactly — then vote across survivors and repair near-misses. Sample count becomes a tunable axis of accuracy; the demonstration pairs are the execution oracle.

e.g. Greenblatt's ~8,000 GPT-4o programs/task (50% public); Berman's evolutionary search (53.6% Pub)

Dominant

Test-time training / per-task adaptation

Briefly fine-tune the model on each task's own demonstrations — a LoRA adapter or full-parameter update, usually with heavy augmentation and an augment–infer–vote loop. This was the decisive lever of 2024, driving the private record from ~33% to 55.5%, and it works best ensembled with symbolic search.

e.g. MIT's LoRA TTT (~6× gain); MindsAI (55.5% season high); the ARChitects' PoE + TTT (Kaggle winner)

Dominant

Frontier reasoning with guided search

Take a general reasoning model and, at test time, search over many candidate chains-of-thought scored by a learned evaluator, spending large inference budgets per task. This is the approach that finally cleared the human-competitive bar — at tens to thousands of dollars per task.

e.g. OpenAI o3 at 87.5% (high-compute), 75.7% (low) — the step-function that triggered ARC-AGI-2

Emerging

Compression / no-pretraining neurosymbolic

Solve a task by minimizing the description length of the puzzle itself, with no pretraining and no search over an external program bank — treating the correct rule as the one that most compresses the demonstrations. A principled outsider to the search-and-adapt mainstream.

e.g. CompressARC (ARC-AGI Without Pretraining), ~20–34% by pure MDL minimization

Open gap

Cheap, human-efficiency solving

The open gap even on a near-solved benchmark: reach o3-level accuracy at anything close to human energy and sample budget. Humans solve ARC-AGI-1 tasks from a few examples for pennies; the machines that match them spend dollars and thousands of samples. Efficiency, not accuracy, is what remains genuinely unsolved here.

e.g. No system yet reaches o3's accuracy at the ARChitects' ~$0.02/task cost

The tradeoffs that shape it

Symbolic search vs neural transduction

hand-built DSL + exact-match search learned neural prediction of the grid

The 2020–2024 arc runs left to right: Icecuber's pure search gave way to neural pipelines, and the induction/transduction paper shows the two families fail on disjoint tasks and only reach human level when ensembled.

Frozen model vs per-task adaptation

one frozen model, in-context only fine-tune on the task's own demos at test time

Per-task adaptation was the single biggest lever of 2024. Why weight-space test-time training so reliably beats in-context learning on the very same examples still has no satisfying theory.

Cheap vs expensive per task

~$0.02/task (the ARChitects, offline) tens to thousands of dollars/task (o3)

ARC-AGI-1 has no cost axis, so o3's accuracy counts regardless of price — but the four-order-of-magnitude spread is exactly why ARC-AGI-2 promoted cost-per-task to a first-class score.

How the record was won

ARC-AGI-1's history is a history of methods finding ways to trade compute, data, or search for score — and of the benchmark holding out far longer than anyone expected. Icecuber's hand-built DSL and brute-force verifier won in 2020 and then held the frontier for three years while large language models stalled below 5%. The template was stable: enumerate programs in a grammar of grid operations, and keep only the ones that replay every demonstration pair exactly.

2024 was the break, and the lever was adaptation, not scale. Test-time training — briefly fine-tuning on each task's own demonstrations — drove the private record from a stalled ~33% to 55.5% (MindsAI, MIT, and the ARChitects), while LLM program synthesis (Greenblatt's thousands of samples, Berman's evolutionary search) proved that sample count against an exact-match verifier is its own axis of accuracy. Then o3 spent enormous test-time compute on guided chain-of-thought search and cleared 87.5% — the first human-competitive score, at tens to thousands of dollars per task.

ARC-AGI-1 is near-solved on accuracy but not on efficiency. Its strongest systems spend far more compute, data, or search than a human, so ARC-AGI-2 adds cost as a scored axis and uses harder compositional tasks to reopen the gap.

What it left behind

Solved on accuracy. Open on efficiency.

The open questions ARC-AGI-1 hands forward, and the gap that reopened the whole project. o3's price tagThe bill: roughly ~$20/task at high efficiency, rising into the thousands at the 172×-compute setting — against about $5 for a human on the same tasks. The accuracy ARC-AGI-1 scored; that four-order spread it did not. is what forced ARC-AGI-2's cost axis; its search-not-understanding character is what the Debates in Part Ⅺ argue over.

1

Did o3 acquire ARC's generalization, or expensively simulate it?

o3 reached 87.5% by searching over many chains-of-thought with a learned evaluator and enormous inference budget. The open question — the crux of the 'o3 is not AGI' critique of 2024–2025 — is whether that reflects the sample-efficient, out-of-distribution generalization ARC was built to measure, or an expensive search that mimics it. By July 2026, newer frontier models also performed strongly on ARC-AGI-2, while ARC-AGI-3 remained much farther from its human baseline. The live question is no longer one score, but whether performance is efficient, transferable, and robust to benchmark changes.
2

Why does test-time training beat in-context learning on the same demos?

Fitting a few gradient steps to a task's own demonstrations reliably outperformed feeding those same demonstrations in-context — a ~6× effect that defined the 2024 season. There is still no satisfying account of the mechanism: better weight-space priors, escaping an attention bottleneck, or implicit search. Without it, we cannot say how far the lever can be pushed or how cheap it can be made.
3

Can a cheap solver match o3's accuracy at human cost?

The ARChitects solved at about two cents per task; o3 spent tens to thousands of dollars for its high score. The efficiency gap — not the accuracy gap — is what ARC-AGI-1 leaves genuinely open, and it is the whole point of ARC-AGI-2's cost axis. Is there a method that reaches the mid-80s at a human-scale sample and energy budget?
4

Why do induction and transduction fail on different tasks?

Explicit program synthesis and direct neural prediction reach human level only when ensembled, because they fail on disjoint sets of tasks. Is the split 'precise/compositional vs. fuzzy/perceptual', a proxy for program length, or something deeper — and could a single model, or a learned router, capture both modes instead of paying for an explicit ensemble?

Figure · The human-vs-AI gap

Where ARC-AGI-1 sits, next to what came after

Best reported AI score against the human baseline for each benchmark generation.

ARC-AGI-12019

87.5%

98%

ARC-AGI-22025

24.03%

100%

ARC-AGI-32026

12.58%

100%

Best AI Human baseline ▨ the gap that remains

ARC-AGI-1 proved the benchmark could be beaten on accuracy — and that beating it cheaply, the way a human does, is a different and unsolved problem. That gap is the reason the story continues.

→ Next: Part Ⅵ · ARC-AGI-2, where cost goes on the board

ARC-AGI-1 five years unbeaten, then a step-function.

Find the rule. Draw the missing grid.

A few examples. One rule to find.

Papers that define ARC-AGI-1.

Reference papers

On the Measure of Intelligence

ARC Prize 2024: Technical Report

Solution papers

ARC 2020 Kaggle 1st-Place Solution (Icecuber)

Getting 50% (SoTA) on ARC-AGI with GPT-4o

The Surprising Effectiveness of Test-Time Training for Abstract Reasoning

Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective

How I got a record 53.6% on ARC-AGI-Pub using Sonnet 3.5 with Evolutionary Test-time Compute

OpenAI o3 Breakthrough High Score on ARC-AGI-Pub

Every major solution, best first.

The score ladder, one row per team

Everything that was tried.

Hand-built DSL + brute-force search

LLM program synthesis (sample-and-verify)

Test-time training / per-task adaptation

Frontier reasoning with guided search

Compression / no-pretraining neurosymbolic

Cheap, human-efficiency solving

The tradeoffs that shape it

How the record was won

Solved on accuracy. Open on efficiency.

Did o3 acquire ARC's generalization, or expensively simulate it?

Why does test-time training beat in-context learning on the same demos?

Can a cheap solver match o3's accuracy at human cost?

Why do induction and transduction fail on different tasks?

Where ARC-AGI-1 sits, next to what came after

ARC Radio

Help improve this guide

Review received