Run Observatory
Twenty-five worlds. One uneven learning landscape.
The official public corpus contains 342 first-run human replays. This index keeps the complete environment-level accounting visible and links every row to the playable task. Search or sort without losing the no-JavaScript table underneath.
The Atlas indexes the complete official aggregate. Full step-by-step frames remain in the official dataset; this site embeds only traces whose artifacts are openly attributable. Official dataset and methodology ↗
Human Learning Atlas
Difficulty has a shape.
Aggregate success ranges from 100% on r11l to 14% on bp35. That spread is the first diagnostic: a benchmark average can hide worlds with very different onboarding cliffs and learning burdens.
Teaching traces
Three traces, three different lessons.
These are evidence-backed reading guides, not reconstructed frame data. They show what to look for before you open the official step-by-step corpus.
10 of 10 participants solved it
- Probe an available action
- Recognize the repeated response
- Reuse the mechanic across six levels
High solvability does not mean zero learning. It means the environment yields a useful rule quickly and consistently.
Inspect the official evidence ↗
8 of 11 solved; two stalled before level 3
- Explore early controls
- Encounter the level-2 bottleneck
- Transfer the discovered mechanic through later levels
Per-level progression distinguishes a hard first abstraction from uniformly hard execution.
Inspect the official evidence ↗
re86Success is not efficiency
5 finished; only 4 reached 100% efficiency
- Acquire the mechanic
- Complete every level
- Compare action count with the median-human baseline
A binary solve hides whether the agent learned economically. RHAE preserves that distinction.
Inspect the official evidence ↗
Failure Museum
A failure is useful when it changes the next experiment.
The diagnosis map separates an observed symptom from its proposed mechanism. Each repair is deliberately framed as a test, not a guaranteed cure.
ObservedExplains a local visual effect but cannot predict the next state
↓
DiagnoseNarrative recognition without a causal transition model
↓
Test nextRecord the transition, make a falsifiable prediction, then probe the smallest disagreement
ObservedImports a familiar game rule despite contradictory observations
↓
DiagnoseA prior is treated as fact rather than a hypothesis
↓
Test nextList competing hypotheses and choose an action whose outcomes separate them
ObservedSolves one level, then rediscovers the same mechanic later
↓
DiagnoseUseful knowledge is not consolidated into durable memory
↓
Test nextStore a compact rule, its evidence, and the conditions under which it applies
ObservedEventually completes the environment after many redundant actions
↓
DiagnosePlanning is disconnected from the human-normalized action budget
↓
Test nextTrack information gained per action and stop experiments once the decision changes
Ablation Board
Put the components on trial.
The 2026 ablation study matters because it refuses a comforting story. Executable persistence alone is not uniformly helpful; simplification usually helps; exact verification is consistently strongest in the main comparisons but costs more.
| Treatment | Persistent model | Simplification | Exact replay | Finding |
| Textual baseline | No | No | No | Can beat a flexible executable model in some settings |
|---|
| Flexible executable model | Yes | No | No | Persistence alone is not reliably beneficial |
|---|
| + scheduled simplification | Yes | Yes | No | Improves 3 of 4 main model-effort settings |
|---|
| + fixed interface and verification | Yes | Yes | Yes | Ranks first in all 4 main settings; uses more resources |
Reproducibility Index
Can another team inspect the claim?
Grades summarize artifact completeness, not scientific importance. “Open code” is only one column; prompts, run artifacts, environment definition, and held-out evaluation determine whether a result can be audited.
| System | Code | Prompts | Run artifacts | Environment | Held-out evidence | Grade |
| EWM | Open | Open | Open | Documented | Not tested | A− |
|---|
| OPINE-World | Paper-linked | Described | Partial | Documented | Not tested | B |
|---|
| EWM ablation | Paper-linked | Treatments specified | Reported | Documented | Not tested | B+ |
|---|
| Milestone #1 agents | Open submissions | Varies | Varies | Competition | Semi-private | B |
|---|
| Human corpus | Dataset | Protocol open | 342 replays | 25 public | First-run humans | A |