# Ai2's StateBench: frontier models fall to 13% or lower when they must track changing facts across long documents

> The Allen Institute for AI released StateBench, a benchmark that tests whether models can follow variables as they change across up to 64,000 updates. Frontier models handle simple lookups but collapse on the hardest updates, and Ai2's small-model experiments point to an architectural reason.

Published Nov 5, 2026 by TechMeasured Staff · AI · Canonical: https://techmeasured.com/ai/ai2-statebench-long-context-state-tracking-frontier-models-13-percent

The **Allen Institute for AI (Ai2)** has released **StateBench**, a benchmark that checks whether language models can keep track of facts that change over a long document, such as a variable reassigned again and again in code, or an object that characters keep moving in a story. Ai2 published it on **November 3** with a tech report, a dataset on Hugging Face and code on GitHub. The headline result: on the longest, hardest problems, frontier models including **Gemini 3.8 Flash, Claude Opus 5.5 and OpenAI's GPT-6 Sol and Luna** dropped to **13% accuracy or lower**, according to Ai2.

Why it matters: long-context claims are usually backed by "find the fact" tests, where the answer sits in one place in a huge input. Real agent work is different. A coding agent reading a long session log, or an assistant following a spreadsheet through hundreds of edits, has to know a value's *current* state after many changes. StateBench separates those two skills, and the gap between them is large.

## Four levels of difficulty

StateBench uses a fixed set of variables and grades problems by the kind of update the model must follow. Ai2 ties the hardest level to a classic hard problem in formal language theory, tracking permutations of five items (S₅). The four levels:

| Level | Example of the update | What the model must do |
|---|---|---|
| R-trivial | A value is set once | Find the original assignment |
| Aperiodic | `x = 7`, later `x = 3` | Keep only the latest value |
| Soluble | `x = x + 5` | Apply a running series of self-referential changes |
| General finite | `x = y`, swaps | Follow values passed between variables by reference |

Unite.AI reports that sequences range from 8 to 64,000 assignments and appear in several formats, including code and spreadsheet cells. Ai2 says the training sequences in its own experiments went up to 256 assignments, while test sequences went up to 64,000.

## What Ai2 found

**Frontier models** did well on the first two levels but dropped sharply on soluble and general finite problems as inputs grew. The 13%-or-lower figure is for the longest general finite problems. Ai2's post presents frontier results in charts and doesn't give a per-model score table, so the post doesn't tell us which model did best.

**Open pretrained models**, including Olmo 3 7B and the Olmo Hybrid 7B, handled r-trivial problems, lost accuracy on aperiodic ones as length grew, and stayed at chance on general finite problems. Format mattered too. Ai2 says its 275M-parameter hybrid and Gated DeltaNet models scored near ceiling on RULER-style r-trivial problems, then fell to near chance when the same problems were written as spreadsheet edits.

**Models trained from scratch** gave the most interesting result. Ai2 trained six 275M-parameter models directly on StateBench: two standard transformers, two hybrids and two Gated DeltaNet models. Gated DeltaNet is a recurrent-style layer that carries a running memory instead of attending over every past token.

- The Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels, even at tens of thousands of assignments.
- Transformers learned the easier levels but degraded on sequences longer than they were trained on, and never learned general finite problems at all.
- A Gated DeltaNet variant with a more flexible memory update stayed near perfect on the hardest level out to about 10 times its training length.

Unite.AI notes that the accompanying paper, by Jackson Petty, William Merrill and Ashish Sabharwal, also includes a proof about which of these problems a fixed-precision Gated DeltaNet can solve. It reads the results as evidence of an architectural limit in transformers.

## Caveats

- **Self-reported.** These are Ai2's results on Ai2's benchmark. No outside group has reproduced them yet.
- **Not how products work.** The frontier models were tested on raw state tracking. Ai2's post doesn't test agents that can write notes, run code or call tools, which is how many products avoid this weakness in practice.
- **Fixed cast.** Unite.AI points out that StateBench assumes a fixed number of variables, so it doesn't model conversations where new entities keep appearing.

## Why developers should care

If you build agents that read long logs, transcripts or edit histories, don't assume a million-token context window means the model knows the current state of things. Ask it to keep explicit state, such as a scratchpad, a database or code that replays the edits, instead of trusting it to work that out over thousands of changes. StateBench is useful for testing exactly this. Ai2 has published the dataset (`allenai/state-bench` on Hugging Face) and the code, which Unite.AI reports is Apache-2.0 licensed.

For model builders, the from-scratch results add to the case for hybrid architectures that mix attention with recurrent layers. It's also worth watching whether the next round of frontier models reports StateBench scores alongside the usual long-context benchmarks.


## Sources

1. [Ai2 — StateBench: Testing how language models track state changes over long documents](https://allenai.org/blog/statebench)
2. [Unite.AI — Ai2 releases StateBench to measure long-context state tracking](https://www.unite.ai/ai2-releases-statebench-to-measure-long-context-state-tracking/)