Learning

Faculty 04/10 · building blockHuman 100.0 · AI 99.9Too close to call +0.13 benchmarks →15 items

At a glance

Picking up new rules or skills from examples or feedback, then applying them.

Learning is picking up new knowledge or skills from examples, instructions or feedback, then applying them correctly to cases not seen before.

Tests of learning have to be new by construction: invented grammars, languages generated on the spot, puzzles whose rule exists only in that puzzle.

Human ref.: 100.0
Best AI: 99.9
Gap: +0.1
Our items: 15

Sub-faculties

The parts of this faculty, what our items cover today, and what is still to come

Sub-faculty	Mirrors	Items	Status
Rule induction from examples Working out a hidden rule from a few examples, then applying it to new cases. How it differs: All the examples come first and the rule is never stated. In learning from feedback nothing is shown first and you learn by trying; in deductive reasoning the rule is given. Compare all four → Mirrors LearningBench	LearningBench	6	Covered
In-context language learning Picking up an unfamiliar notation from a short description and using it straight away. Mirrors EphLangBench	EphLangBench	6	Covered
Learning from feedback Improving from right-or-wrong feedback alone, with no explanation. How it differs: Nothing is shown up front: you have to choose, see how it turned out, and adjust. In rule induction all the examples come first; here the evidence comes from your own choices, one at a time. Compare all four → Mirrors Binz & Schulz (2023)	Binz & Schulz (2023)	3	Covered

Human vs AI

Best published result on each side, 0–100

Human ref.

100.0

Best AI

99.9

Humans. Members of the public solve ARC-AGI-3 games. In artificial-grammar studies, people classify new strings above chance without being able to state the rule.

AI. Results depend heavily on setup: the same model can score far higher with a custom harness than with the standard one. On rules never seen before, most models struggle.

Human ref. 100.0: ARC-AGI-3's human result: members of the public, tested in person with no instructions, solved every one of its puzzle games (ARC Prize). About the test →
Best AI 99.9: The best ARC-AGI-3 score, GPT-6 Astra with a custom harness that keeps its reasoning between moves. On the standard harness, the top score is 62.7. About the test →

Human referenceOur estimateBest published AILabel = human − AI, points

Example items

Specimens written for this page. Live items are similar but never published.

Learn an alien grammarspecimen · not scored

These strings follow one hidden rule. Decide whether each new string follows it.

(a) Study strings

KL KRRL KZL KZRRL KRRZL KRRRRL KZZL

(b) Test strings: follows / breaks

KRRZZL □ □KRL □ □ZRRL □ □KZRRZL □ □KRRZ □ □KRRRL □ □

Figure 1. An artificial-grammar item. The rule is never stated; it must be induced from seven examples. Benchmark: LearningBench. Answer: Start with K, end with L, and R only ever comes in pairs. Follows: KRRZZL, KZRRZL.

LearningBench benchmark page →

Speak Vornspecimen · not scored

Evaluate each expression. Answer in ordinary numbers.

(a) Language definition

In Vorn the operator comes after its two numbers: dok adds, fen subtracts.

(b) Worked examples

4 3 dok = 7

9 2 fen = 7

2 3 dok 4 fen = 1

6 1 fen 3 dok =

8 2 3 dok fen =

Figure 2. An invented-language item. Syntax is defined on the spot, so nothing can be recalled from training. Benchmark: EphLangBench. Answer: 8 (6 − 1 = 5, then 5 + 3) and 3 (2 + 3 = 5, then 8 − 5).

EphLangBench benchmark page →

Find the better machinespecimen · not scored

Choose a machine 10 times. Each pull shows only whether it paid.

(a) Each pull shows only whether it paid

Left machine
+1 point

Right machine
nothing

Figure 3. A two-machine choice. The left machine paid on the first pull, which tempts people to stay with it. Benchmark: Binz & Schulz (2023). Answer: The right machine pays on 7 of 10 pulls, the left on 3.

Binz & Schulz (2023) benchmark page →

Benchmarks

3 rows

Benchmark	Origin	Items	Best AI	Humans
LearningBench	Kaggle × Google DeepMind hackathon, grand prize, 2026	6	0.85	~60%
EphLangBench	Kaggle × Google DeepMind hackathon, Learning track winner, 2026	6	89%	—
Binz & Schulz (2023)	Binz & Schulz (2023), PNAS, 2023	3	2.72	3.24

In everyday life

Picking up a board game from one demo round
Inferring a colleague's filing system
Learning a new app without the manual