StateBench shows frontier state tracking floors at 13 percent
Transformers trained from scratch never learned the hardest state-tracking level, and Allen AI's new StateBench now quantifies the gap: frontier models floor at 13 percent on it.

| Frontier models | 7B open pretrained | 275M from-scratch | |
|---|---|---|---|
| General finite accuracy | 13% or lower | chance | not learned |
| Max training context | 64,000 tokens | 256 assignments | |
| Max test context | beyond 64,000 tokens | 64,000 assignments |
Allen AI released StateBench on November 3, 2026, a four-level benchmark for tracking how entities change state across long documents. The levels range from immutable assignment through S₅ permutation composition.
The hardest level, general finite, requires assignment by reference across a chain of swaps with no shortcut. Frontier models—Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and Luna—dropped to 13 percent or lower on the longest sequences. Open pretrained models at 7B parameters stayed at chance.
The constraint is architectural. Transformers trained from scratch at 275M parameters on StateBench never learned the general finite level. Gated DeltaNet, with a flexible memory update, generalized near-perfect to roughly 10× its 256-assignment training length.
The document does not prove that Gated DeltaNet with non-negative eigenvalues handles all soluble tasks, nor does it detail the specific change that lifts SSMs out of TC⁰. Fixed-precision arithmetic assumptions are stated but not stress-tested against floating-point drift at 64,000-token lengths.
Dynamic entity introduction—new variables entering mid-sequence—falls outside the current framework entirely. For a team choosing a long-context backbone, the 13 percent floor means no frontier model can be trusted to follow a chain of object swaps past a few hundred steps without a recurrent or hybrid layer.
The draft repeated the 13 percent finding twice and listed the same transformer limitation in two paragraphs. The rewrite consolidates the architectural contrast into one block, moves the benchmark's scope to the opening, and folds the caveats into a single paragraph so the practical takeaway lands last without restating the floor.
After arxiv.org. We did not report this. The pictures, if any, are theirs.


