Binz & Schulz (2023)
View original ↗At a glance
- Tests
- Learning from feedback (Learning): finding the better option from wins and losses alone, with no explanation.
- How it differs
- Nothing is shown up front: you have to choose, see how it turned out, and adjust. In rule induction all the examples come first; here the evidence comes from your own choices, one at a time. Compare all four →
- How
- Two options pay out with different, hidden chances. Each choice shows only what it paid; the better option has to be learned from that feedback.
- Best AI
- 2.72
- Humans
- 3.24
- Our items
- 3
- Learning gap, overall
- +0.1
Example item
A specimen written for this page. Live items are similar but never shown, so posted answers can't spoil them.
Choose a machine 10 times. Each pull shows only whether it paid.
(a) Each pull shows only whether it paid
+1 point
nothing
Human vs AI
What the benchmark reports, and where the faculty stands overall
Best published AI
2.72
GPT-3's average regret on two-machine choices (people: 3.24; lower is better)
Humans
3.24
average regret, people (lower is better)
Learning overall · faculty reference, 0–100
Measured on other tests, not on Binz & Schulz (2023).
Human: ARC-AGI-3, members of the public. AI: ARC-AGI-3, GPT-6 Astra with custom harness.
How we mirror it
Twenty pulls between two machines per item. One pays out more often, but its first pull is a miss, so early luck is misleading.
Payouts are fixed in advance, so everyone faces the same luck. Every choice is stored; the score is how often the better machine is chosen in the last ten pulls.
Items are original, versioned, and assigned at random, so a posted answer spoils only one variant.
What we record