Can AI systems acquire unfamiliar rules through exploration? We test them in verifiable Alien Worlds, whose rules are executable and conflict with familiar knowledge.
Three independent trajectories per system and sandbox, ranked by Best@3, the highest M4. Every score is the mean of three answers per question.
M0 and M4 come from the Best@3 trajectory; Mean@3 and Worst summarise all three. O@M0 and A4+O supply the complete rule set before exploration or after it. Each system runs at its highest reasoning setting. Aux counts the rules reported correctly in AlienCode and the rate of correct refusal on unprovable theorems in AlienLogic; it never enters M4.
From left to right the system takes a larger part in designing the experiments. The first two columns receive no environment feedback.
Autonomous is the Best@3 trajectory, and hindsight replays that trajectory's probes. Each control condition is one trajectory per system.
Each curve is a system's Best@3 trajectory. Click systems to compare; the badge gives the retained gain M4 − M0.
Evaluating exploration ability is difficult because two requirements are in tension.
A correct answer may simply have been memorised, and contamination cannot be ruled out for a black-box model.
Checking a genuinely new result can take expert proof checking, costly experiments, or years. Without it, a real discovery and a confident mistake look identical.
Alien Worlds are executable, so every answer is checked exactly, and they conflict with familiar knowledge, so recall misleads rather than helps: SHATTER multiplies, EMIT(100) prints 127, and no primitive adds.
Two deterministic, executable sandboxes. Every altered rule is covered by at least one held-out task, and no LLM acts as judge.
A small calculation language whose familiar-looking operators follow hidden semantics. Programs are graded by an interpreter on private inputs.
A first-order natural-deduction system with patched inference rules. A proof checker verifies every proof, and an unprovable theorem scores only when the system declines to prove it.
The manual is familiar; the world is not. Try a probe.
One continuous history, four rounds of probing, and no weight updates. After every round, a tool-disabled copy reports its rules and answers the held-out tasks.
Whether exploration works, how discovering a rule relates to using it, and how reliably exploration succeeds.
Four rounds lift AlienCode from at most 15.7% to 89.0%, while the same turns without feedback stay at or below 11.0%. Designing the probes matters more than receiving their results.
Fig. 4 · Each dot is one system; the line marks the median. Shaded columns receive no environment feedback.
Fig. 5 · A system's rank in one sandbox barely predicts its rank in the other (Spearman 0.35).
AlienLogic is limited by discovery and AlienCode by use. Two keystone rules decide most of AlienCode, and stating a rule still does not guarantee using it.
Fig. 6 · Open-book answering before exploration (hatched), autonomous exploration alone (solid), and open-book answering after it (outlined).
Fig. 7 · Triangles mark the milestone at which the rule report first states R15 (CARVE slice shift) and R14 (PLUCK index shift) correctly.
Fig. 8 · Tasks whose required rules the M4 report states correctly (filled) are still solved only 73.4% of the time.
Repeated answers barely move, while one system's trajectories end tens of points apart. The largest gain arrives in a single round, and later rounds can undo it.
Fig. 9 · Dots are the three trajectories' M4; the grey halo is one standard deviation of that trajectory's three answers.
Fig. 10 · Bubble area is the change from the previous milestone; the solid bubble marks the largest step, and a red ring marks a fall.
The same design can evaluate exploration wherever unfamiliar rules can be made executable and their consequences verified exactly.
The tasks are not public. To have a model evaluated, contact us and we will run it under the same protocol as the leaderboard.