The Allen Institute for AI (Ai2) has released StateBench, a new benchmark that tests how well language models keep track of things that change over the course of a long document. Announced on 3 November 2026, it found that leading models, including Gemini 3.8 Flash, Claude Opus 5.5 and GPT-6 Sol and Luna, handle simple cases well but fall to 13% accuracy or lower on the hardest and longest problems.
The finding
"State tracking" means remembering the current situation as it changes: the value of a variable in code after several updates, or who is holding an object in a story after it has been passed around. Some cases only need the latest update. Harder ones need the model to follow a whole chain of changes, where missing a single step gives the wrong answer.
All four frontier models did well on the two easier levels. On the two harder levels, their accuracy dropped sharply as the inputs grew longer. Ai2 suggests one reason: at those levels many more steps can affect the final answer, so there is less room to miss an update.
How it was done
StateBench sorts problems into four levels, each allowing a more complex kind of update:
- R-trivial: a value is set once and never changes, like a birth date.
- Aperiodic: a value can be replaced, so only the latest one counts, like a thermostat setting.
- Soluble: each update builds on the current value, like a running game score.
- General finite: one variable can take another's value, as in tracking a ball under five cups that keep being swapped.
To keep the comparison fair, the team builds each hard problem first and derives the three easier versions from it, with the same number of variables and updates. The levels come from Krohn-Rhodes theory, a branch of algebra.
Results
- Open pretrained models: Ai2 tested its Olmo 3 (a transformer) and Olmo Hybrid (which adds Gated DeltaNet layers), plus smaller models. Almost all did well on the easiest level, lost accuracy on the second as inputs grew, and stayed at chance on the hardest. The way a problem was presented also mattered: some small models scored near the top in one format and near chance on the same problem framed as spreadsheet edits.
- Models trained on the task: small models of 275 million parameters were trained on sequences of up to 256 updates and tested on up to 64,000. Gated DeltaNet and hybrid models stayed almost perfect on the first three levels at every length tested. Transformers degraded on longer inputs and never learned the hardest level.
Ai2 concludes that for recurrent and hybrid designs the weakness comes from training rather than the architecture, while transformers hit a limit even with direct training.
Limits stated by the authors
The benchmark assumes a fixed number of variables, while real conversations keep introducing new people and objects. Some theory questions about where Gated DeltaNet fits in the hierarchy are still open. Ai2 has published a technical report, the data and the code.
Announced by Ai2 on its official blog: read the announcement
