Huang et al. (2025)
View original ↗At a glance
- Tests
- Working memory (Memory): holding a few things in mind while updating or rearranging them, with nothing left to look at.
- How
- Models must keep a secret number, or shuffle four numbers mentally, without writing anything down. The information is never in front of them, so they can't simply look back at it.
- Best AI
- 26%
- Humans
- —
- Our items
- 6
- Memory gap, overall
- +21.4
Example item
A specimen written for this page. Live items are similar but never shown, so posted answers can't spoil them.
Digits appear one at a time. When they stop, type them in reverse order, last one first.
(a) Shown one at a time (0.9 s each)
Human vs AI
What the benchmark reports, and where the faculty stands overall
Best published AI
26%
best model keeping numbers in mind without writing them down
Humans
—
This benchmark doesn't report a human figure.
Memory overall · faculty reference, 0–100
Measured on other tests, not on Huang et al. (2025).
Human: LoCoMo F1 (Maharana et al., 2024). AI: LoCoMo F1, Claude Sonnet + memory system.
How we mirror it
Digit span and letter span, the classic measure of working memory: digits or letters appear one at a time and disappear, then you type them back, in order or last one first. Three rounds per item, each longer than the last.
Nothing stays on screen, so the sequence has to be held in mind: the same condition the paper puts models in. We store what was typed in each round, not just right or wrong.
Items are original, versioned, and assigned at random, so a posted answer spoils only one variant.
What we record