The Allen Institute for AI (Ai2) has released StateBench, a benchmark that checks whether language models can keep track of facts that change over a long document, such as a variable reassigned again and again in code, or an object that characters keep moving in a story. Ai2 published it on November 3 with a tech report, a dataset on Hugging Face and code on GitHub. The headline result: on the longest, hardest problems, frontier models including Gemini 3.8 Flash, Claude Opus 5.5 and OpenAI's GPT-6 Sol and Luna dropped to 13% accuracy or lower, according to Ai2.
Why it matters: long-context claims are usually backed by "find the fact" tests, where the answer sits in one place in a huge input. Real agent work is different. A coding agent reading a long session log, or an assistant following a spreadsheet through hundreds of edits, has to know a value's current state after many changes. StateBench separates those two skills, and the gap between them is large.
Four levels of difficulty
StateBench uses a fixed set of variables and grades problems by the kind of update the model must follow. Ai2 ties the hardest level to a classic hard problem in formal language theory, tracking permutations of five items (S₅). The four levels:
| Level | Example of the update | What the model must do |
|---|---|---|
| R-trivial | A value is set once | Find the original assignment |
| Aperiodic | x = 7, later x = 3 |
Keep only the latest value |
| Soluble | x = x + 5 |
Apply a running series of self-referential changes |
| General finite | x = y, swaps |
Follow values passed between variables by reference |
Unite.AI reports that sequences range from 8 to 64,000 assignments and appear in several formats, including code and spreadsheet cells. Ai2 says the training sequences in its own experiments went up to 256 assignments, while test sequences went up to 64,000.
What Ai2 found
Frontier models did well on the first two levels but dropped sharply on soluble and general finite problems as inputs grew. The 13%-or-lower figure is for the longest general finite problems. Ai2's post presents frontier results in charts and doesn't give a per-model score table, so the post doesn't tell us which model did best.
Open pretrained models, including Olmo 3 7B and the Olmo Hybrid 7B, handled r-trivial problems, lost accuracy on aperiodic ones as length grew, and stayed at chance on general finite problems. Format mattered too. Ai2 says its 275M-parameter hybrid and Gated DeltaNet models scored near ceiling on RULER-style r-trivial problems, then fell to near chance when the same problems were written as spreadsheet edits.
Models trained from scratch gave the most interesting result. Ai2 trained six 275M-parameter models directly on StateBench: two standard transformers, two hybrids and two Gated DeltaNet models. Gated DeltaNet is a recurrent-style layer that carries a running memory instead of attending over every past token.
- The Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels, even at tens of thousands of assignments.
- Transformers learned the easier levels but degraded on sequences longer than they were trained on, and never learned general finite problems at all.
- A Gated DeltaNet variant with a more flexible memory update stayed near perfect on the hardest level out to about 10 times its training length.
Unite.AI notes that the accompanying paper, by Jackson Petty, William Merrill and Ashish Sabharwal, also includes a proof about which of these problems a fixed-precision Gated DeltaNet can solve. It reads the results as evidence of an architectural limit in transformers.
Caveats
- Self-reported. These are Ai2's results on Ai2's benchmark. No outside group has reproduced them yet.
- Not how products work. The frontier models were tested on raw state tracking. Ai2's post doesn't test agents that can write notes, run code or call tools, which is how many products avoid this weakness in practice.
- Fixed cast. Unite.AI points out that StateBench assumes a fixed number of variables, so it doesn't model conversations where new entities keep appearing.
Why developers should care
If you build agents that read long logs, transcripts or edit histories, don't assume a million-token context window means the model knows the current state of things. Ask it to keep explicit state, such as a scratchpad, a database or code that replays the edits, instead of trusting it to work that out over thousands of changes. StateBench is useful for testing exactly this. Ai2 has published the dataset (allenai/state-bench on Hugging Face) and the code, which Unite.AI reports is Apache-2.0 licensed.
For model builders, the from-scratch results add to the case for hybrid architectures that mix attention with recurrent layers. It's also worth watching whether the next round of frontier models reports StateBench scores alongside the usual long-context benchmarks.
