The desk
Die Brief

DIE.BRIEF

Hardware desk
Models SINGLE-SOURCE

StateBench shows frontier state tracking floors at 13 percent

The Die Brief Desk, after arxiv.org

Transformers trained from scratch never learned the hardest state-tracking level, and Allen AI's new StateBench now quantifies the gap: frontier models floor at 13 percent on it.

Picture from huggingface.co
Picture from huggingface.co
Frontier models7B open pretrained275M from-scratch
General finite accuracy13% or lowerchancenot learned
Max training context64,000 tokens256 assignments
Max test contextbeyond 64,000 tokens64,000 assignments

Allen AI released StateBench on November 3, 2026, a four-level benchmark for tracking how entities change state across long documents. The levels range from immutable assignment through S₅ permutation composition.

The hardest level, general finite, requires assignment by reference across a chain of swaps with no shortcut. Frontier models—Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and Luna—dropped to 13 percent or lower on the longest sequences. Open pretrained models at 7B parameters stayed at chance.

The constraint is architectural. Transformers trained from scratch at 275M parameters on StateBench never learned the general finite level. Gated DeltaNet, with a flexible memory update, generalized near-perfect to roughly 10× its 256-assignment training length.

The document does not prove that Gated DeltaNet with non-negative eigenvalues handles all soluble tasks, nor does it detail the specific change that lifts SSMs out of TC⁰. Fixed-precision arithmetic assumptions are stated but not stress-tested against floating-point drift at 64,000-token lengths.

Dynamic entity introduction—new variables entering mid-sequence—falls outside the current framework entirely. For a team choosing a long-context backbone, the 13 percent floor means no frontier model can be trusted to follow a chain of object swaps past a few hundred steps without a recurrent or hybrid layer.

The draft repeated the 13 percent finding twice and listed the same transformer limitation in two paragraphs. The rewrite consolidates the architectural contrast into one block, moves the benchmark's scope to the opening, and folds the caveats into a single paragraph so the practical takeaway lands last without restating the floor.

After arxiv.org. We did not report this. The pictures, if any, are theirs.

November 04, 2026 · 2 min
Copy link Share to X