LoCoMo

Maharana et al., 2024

What it measures

A published test we don't mirror: it supplies a reference figure, not test items

LoCoMo tests memory over very long conversations: about 300 turns and 9,000 tokens each, spread over up to 35 sessions.

Models answer questions about what was said earlier, including when things happened and why. Scores are F1: how much of the right answer a reply contains.

Example question

Written for this page in the style of the test, not one of its questions, so the test set stays unspoiled

Weeks apartspecimen · not scored

Read the conversation, then answer.

(a) Conversation excerpt

Session 3, 12 May. Maya: I finally did it, I adopted a greyhound! She's called Pepper.

… five sessions about work, a move and a cousin's wedding …

Session 9, 2 July. Tom: How's the dog? Maya: Pepper's great. We just got back from her first trip to the beach.

(b) Question

How long had Maya had her dog when they went to the beach?

Figure 1. A temporal question: the answer joins two facts said sessions apart and the dates they were said. In the style of LoCoMo. Answer: About seven weeks (12 May to 2 July).

Human vs AI

The faculty-level figures we take from it, 0–100

MemoryGap +21.4

Humans

87.9

LoCoMo F1 (Maharana et al., 2024)

Best published AI

66.5

LoCoMo F1, Claude Sonnet + memory system

These are third-party results on different questions from ours. Where the numbers come from

All reference tests →