StateBench: Frontier Models Fall to 13% on State Tracking

Agentic Benchmark · ax-trend issue #3810 · Released November 3, 2026

StateBench title card with the 13 percent frontier result

Ai2 released StateBench on November 3, 2026, on Hugging Face, together with a technical report, data and code. The benchmark asks a narrower question than most long context tests: when information changes inside a long document, can a model keep track of what the current value actually is?

The answer separates cleanly by difficulty. Frontier models handled the easy tiers, then fell to 13% or less on the longest tasks in the hardest tier. That gap is the point of the design. Finding a passage in a long document is retrieval, and retrieval turns out to be the easiest part. Following a chain of updates, where each new value depends on a value that was itself updated earlier, is a different ability, and it is the one that long agent runs and evolving codebases actually require.

Frontier accuracy, longest general finite tasks13% or less
Difficulty tiers4
Small model training length256 assignments
Small model test length64,000 assignments, 250x training

Four Tiers Built From One Task

Diagram of the four StateBench tiers from R-trivial to general finite
The four tiers in order of difficulty, from a value that never changes to an update that reads another variable.

The four tiers are grounded in Krohn-Rhodes theory. R-trivial covers values that do not change, and aperiodic covers direct reassignment to a new value. Both sit on the easier side. Soluble updates a value from its own current value, as in x = x + 5. General finite updates a value by referencing the current value of a different variable, as in x = y, and it is the hardest tier.

The comparison is controlled in a way many benchmarks are not. All four tiers derive from a single task that shares the same number of variables, the same number of distinct values and the same assignment positions. Only the type of assignment operation changes. When accuracy collapses between tiers, the cause is therefore the update logic itself, not a longer input or a larger variable set hiding inside the harder test.

Frontier Model Results

Ai2 tested Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna. All four performed well on R-trivial and aperiodic tasks. On soluble and general finite tasks, accuracy fell sharply as the input grew longer, reaching 13% or less on the longest general finite tasks.

For anyone selecting a model for long running agents, this is a direct counterexample to a common shortcut. A strong score on a retrieval style long context benchmark says the model can find relevant text. StateBench shows that the same model can still lose the thread once the task requires applying a long sequence of dependent updates correctly. The two skills need separate evidence, and until recently only one of them had a dedicated test.

Pretrained Open Models and the Length Boundary

Ai2 also tested two pretrained open models trained on the same data: Olmo 3, a 7B transformer, and Olmo Hybrid, a 7B model that includes Gated DeltaNet layers. On the general finite tier, both stayed close to chance. Neither model had seen contexts longer than 64,000 tokens in training, while the test extended beyond that length, so part of the result is a length generalization boundary rather than a pure architecture verdict.

That boundary matters for interpretation. The pretrained results show that current open models do not acquire hard state tracking as a side effect of ordinary pretraining at this scale. They do not, on their own, show which architecture would win if both were trained directly for the task. Ai2 ran that second experiment separately, and its outcome is different.

Task Format Changes the Result

How a task is written down changed the scores as much as model choice did. The 275M hybrid and Gated DeltaNet models scored close to ceiling on R-trivial tasks presented in the RULER format. When Ai2 presented the same task as spreadsheet editing, the same models fell to chance level.

This is a warning for benchmark consumers and for agent builders at the same time. A model that tracks state reliably in one surface format may fail when the identical state is presented through a different interface, such as a spreadsheet, a tool schema or a structured document. Evaluations that use a single format can overstate how robust the underlying ability is, and production systems inherit that fragility when their input format differs from the benchmark format.

Small Models Trained Directly on the Task

The most striking part of the release is the training experiment. Ai2 trained six 275M models directly on StateBench tasks: two transformers, two hybrids and two Gated DeltaNet models. Training covered up to 256 assignments, and testing extended to 64,000 assignments, which is 250 times the training length.

The Gated DeltaNet and hybrid models generalized close to ceiling on the easier three tiers, all the way to the longest sequences. The transformers never fully learned the hardest tier. One detail points at the mechanism: the Gated DeltaNet variant with flexible memory updates stayed close to ceiling on the hardest tier up to about 10 times the training length. A memory that can be updated flexibly appears to carry state in a way that repeated attention over a growing document does not match on this task type.

The practical reading is architectural, not just empirical. If a workload is dominated by dependent state updates, architecture choice and memory update design may matter more than raw context length. A smaller recurrent or hybrid model, trained for the pattern, generalized hundreds of times beyond its training length where much larger general models failed.

Limits and Open Questions

Every result in this article is an Ai2 measurement on synthetic tasks. Performance on StateBench does not transfer directly to real codebases or general documents. The benchmark also assumes a fixed number of variables, so settings where new entities keep appearing, such as long conversations, are not yet covered.

Two questions remain open in Ai2 own account. How far a Gated DeltaNet model with restricted updates can go across the four tiers is still theoretically unresolved, and the existing results hold within a proof scope that assumes fixed precision arithmetic and rounding. The fixed variable setting is the other gap: real agent work introduces new files, new objects and new people over time, and a benchmark with a fixed variable set cannot measure tracking under that kind of growth.

Even with those limits, StateBench changes what a careful model evaluation needs to include. Context length and retrieval scores remain useful, but they no longer stand in for state tracking. A team running long agents now has a public test, with data and code, that measures the update following ability directly, and a baseline result showing that the largest current models score 13% or less where that ability is tested hardest.

Related Reading

For a related argument about maintaining long tasks without long context, see the related report on frequent compression for long tasks. For earlier Ai2 open model work, see the AstaBrief release report.

FAQ

What is StateBench?

StateBench is a benchmark Ai2 released on Hugging Face on November 3, 2026, with a technical report, data and code. It measures how well a model tracks the current state of information as that information changes inside a long document, across four tiers of difficulty grounded in Krohn-Rhodes theory.

How do frontier models perform on StateBench?

Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna performed well on the two easier tiers. On the soluble and general finite tiers their accuracy fell sharply as inputs grew longer, dropping to 13% or less on the longest general finite tasks.

Why do small recurrent models generalize further on StateBench?

In Ai2 tests, 275M Gated DeltaNet and hybrid models trained on up to 256 assignments generalized close to ceiling on the easier three tiers at sequences up to 64,000 assignments. A Gated DeltaNet variant with flexible memory updates stayed close to ceiling on the hardest tier up to about 10 times the training length, while transformers never fully learned that tier.

What are the limits of the StateBench results?

The results are Ai2 measurements on synthetic tasks, so they do not transfer directly to real codebases or general documents. The benchmark also assumes a fixed number of variables, so settings where new entities keep appearing, such as long conversations, are not yet covered.

Sources

Agentic BenchmarkOpen ModelsPerformanceAX Trend