Problem solving

Faculty 09/10 · compositeHuman 83.7 · AI 81.9Too close to call +1.82 benchmarks →14 items

At a glance

Combining faculties to solve problems, from novel puzzles to common sense.

Problem solving is a composite faculty: it combines perception, reasoning, learning and the rest to solve problems, from novel puzzles to everyday common-sense situations.

Two very different tests sit here: abstract grid puzzles with a new rule each time, and everyday trick questions where the arithmetic is a red herring.

Human ref.: 83.7
Best AI: 81.9
Gap: +1.8
Our items: 14

Sub-faculties

The parts of this faculty, what our items cover today, and what is still to come

Sub-faculty	Mirrors	Items	Status
Everyday traps Using common sense on questions built to punish the obvious answer. Mirrors SimpleBench	SimpleBench	7	Covered
Abstract rule induction Finding the rule behind a visual puzzle and producing the answer. How it differs: Like rule induction from examples (Learning), the rule is found from examples; here it is a change to a picture that you then have to carry out, which is why it counts as problem solving. Compare all four → Mirrors ARC-AGI	ARC-AGI	7	Covered

Human vs AI

Best published result on each side, 0–100

Human ref.

83.7

Best AI

81.9

Humans. Untrained adults do well on everyday trick questions, and a human panel solves every ARC-AGI-2 puzzle.

AI. The best models are close to people on everyday trick questions, but don't solve every abstract grid puzzle.

Human ref. 83.7: Share of SimpleBench's trick questions answered correctly by nine people without special training (SimpleBench's own baseline). About the test →
Best AI 81.9: The same questions answered by the best model on the SimpleBench leaderboard, Claude Fable 5. About the test →

Human referenceOur estimateBest published AILabel = human − AI, points

Example items

Specimens written for this page. Live items are similar but never published.

Find the rulespecimen · not scored

Each example turns the left grid into the right one. Paint the answer for the test grid.

(a) Worked examples

→

(b) Test input

Figure 1. A grid-induction item. Two worked examples define a rule the participant must apply to a new grid. Benchmark: ARC-AGI. Answer: Gravity: every blue cell falls to the bottom of its column, stacking on others.

ARC-AGI benchmark page →

Mind the trapspecimen · not scored

An everyday question with a twist.

(a) Question

1. Jo holds a full, lidless cup of coffee in a lift. On the way up four floors, Jo turns the cup upside down for one second. When the doors open, how much coffee is in the cup?
(a) Full(b) About half(c) Most of it(d) Little or none

Figure 2. A common-sense item. The arithmetic-looking detail (four floors) is irrelevant; physics decides. Benchmark: SimpleBench. Answer: Little or none: it poured out when the cup was turned over.

SimpleBench benchmark page →

Benchmarks

2 rows

Benchmark	Origin	Items	Best AI	Humans
ARC-AGI	ARC Prize Foundation, 2019	7	95.0%	100%
SimpleBench	SimpleBench, 2024	7	81.9%	83.7%

In everyday life

Fixing something with whatever is at hand
Noticing a question is a trick
Finding the pattern in a messy situation