Strachan et al. (2024)
View original ↗At a glance
- Tests
- Theory of mind (Social cognition): working out what someone else believes or knows, even when it differs from what you know.
- How
- Short stories followed by questions: where will someone look for an object moved while they were away, did a speaker realise they had said something hurtful, why did a character say something untrue.
- Best AI
- 4 of 5
- Humans
- —
- Our items
- 3
- Social cognition gap, overall
- +4.4
Example item
A specimen written for this page. Live items are similar but never shown, so posted answers can't spoil them.
A short story, then a question.
(a) Question
- 1. Mia leaves her sandwich on the bench and goes to wash her hands. While she's gone, a friend moves it to the shelf. When Mia comes back, where will she look for her sandwich first?(a) On the shelf(b) On the bench(c) In her bag(d) She'll ask the friend
Human vs AI
What the benchmark reports, and where the faculty stands overall
Best published AI
4 of 5
theory-of-mind tests where GPT-4 matched or beat people
Humans
—
This benchmark doesn't report a human figure.
Social cognition overall · faculty reference, 0–100
Measured on other tests, not on Strachan et al. (2024).
Human: Social IQa (Sap et al., 2019). AI: CogToM, GPT-5.1 (a different test).
How we mirror it
New stories in the same formats: a false belief and what one person thinks another believes, a faux pas, a white lie and a double bluff. The classic versions (Sally-Anne and others) are in every model's training data, so ours are written fresh.
Each item has two questions; both must be right.
Items are original, versioned, and assigned at random, so a posted answer spoils only one variant.
What we record