Latent Sparse Attention: 72.4x Prefill at 10M Tokens

Prefrontal AI open research · ax-trend issue #3742 · Released November 3, 2026 · Paper arXiv:2610.42147

Latent Sparse Attention title card: 72.4x prefill at 10M tokens
Overview: hierarchical search over meaning clusters, with prefill speedups reported at 10M tokens

Prefrontal AI released Latent Sparse Attention, or LSA, as open research on November 3, 2026, publishing a paper, code, and a technical blog post together. The headline result is an attention layer prefill speedup of 72.4x over full attention at 10M tokens on H200 hardware, measured after distilling LSA into an existing Qwen-3.8-27B model. End to end serving on the same setup was 9.2x faster, and knowledge and reasoning scores stayed within 2 points of the original model in the team's own tests.

The claim underneath those numbers is more specific than a faster kernel. Prefrontal AI argues that at very long context lengths, the expensive part of sparse attention is no longer the attention math itself. At 10M tokens, the team says 98% of sparse attention time goes to indexing, the work of finding which tokens are worth looking at. LSA attacks that search step directly by organizing keys into a hierarchy of meaning based clusters, so finding relevant tokens becomes a near logarithmic search instead of a scan whose cost grows quadratically.

The Indexing Problem Behind Sparse Attention

Full attention compares every query with every key, so its cost grows as O(T2) with sequence length T. That quadratic curve is tolerable at thousands of tokens and punishing at millions, which is the range agent workflows are now moving into as context windows pass 1M tokens. Sparse attention methods cut the number of tokens each query actually attends to, and that part of the problem is well understood.

The less visible problem is selection. Before a sparse method can attend to a small subset, it has to find that subset, and at extreme lengths the finding can cost as much as the attending it was meant to save. Prefrontal AI puts that cost at 98% of sparse attention time at 10M tokens. Read that way, the release is less about a new sparsity pattern and more about a new index structure, one designed so that search cost falls along with attention cost instead of replacing one quadratic term with another.

Diagram of Latent Sparse Attention: cluster keys by meaning, score clusters, search inside relevant groups, then attend sparsely at O(T log T) cost
Method flow: cluster first, score the clusters, and search only inside the groups a query actually needs

How Latent Sparse Attention Works

LSA clusters keys hierarchically in latent space, grouping them by meaning rather than by position in the sequence. A query first scores the top level clusters, then descends only into the clusters that scored well and searches inside them. Position based blocking, the usual way of chunking a long sequence, cannot do this, because tokens that belong together semantically can sit millions of positions apart in an agent transcript or a long document.

With full hierarchy in place, Prefrontal AI states that attention cost falls from quadratic O(T2) to near quadratic O(T log T). The practical consequence is in the exponent, not in any single benchmark. A method whose cost grows as T log T keeps improving relative to full attention as context grows, so the reported gap at 10M tokens should be read as a point on a widening curve. It also means the advantage at ordinary lengths, tens of thousands of tokens, would be far smaller, a scope limit the headline number does not show.

Prefill and Serving Speedups in D-LSA

The speed results come from D-LSA, the variant distilled into already trained models with a short distillation step. On Qwen-3.8-27B at 10M tokens, attention layer prefill ran 72.4x faster than full attention on H200 and 42.9x faster on B300, while end to end serving improved 9.2x. The gap between the layer level and end to end figures is expected and worth keeping in view. Prefill of the attention layers is only part of serving cost, so a 72.4x layer result compresses to a single digit system result even when it works exactly as reported.

Bar chart of speedups versus full attention: Qwen H200 prefill 72.4x, Qwen B300 prefill 42.9x, GLM prefill 22.7x, Qwen serving 9.2x
Reported speedups versus full attention at 10M tokens, from Prefrontal AI measurements

A second test replaced the built in sparse attention of GLM-5.3-Flash with LSA. Prefill improved 22.7x and end to end speed improved 2.2x. That comparison matters more than the full attention baseline in one respect. GLM already ships a sparse mechanism, so the result suggests LSA's index structure beats an existing production sparse design on its own workload, not only the dense baseline everyone expects to beat. Both results remain single source measurements, a point returned to below.

Accuracy Results and the Places They Slip

On knowledge and reasoning benchmarks, Prefrontal AI reports that both models stayed within 2 points of their originals after LSA was applied. The individual scores are close enough to support that summary for this category of test.

BenchmarkOriginalWith LSA
MMLU-Pro83.283.0
GPQA-D88.487.9
AIME2598.397.5
MATH50098.099.0

Long context tests tell a less uniform story, and the release states the losses plainly. Qwen held parity on RULER out to 10M tokens, but its LongBench v2 score fell from 70.0 to 65.0, a 5 point drop. GLM lagged by up to about 8 points on RULER in the 2M to 5M token range. The pattern is informative. Scores that depend on finding and combining distant evidence are exactly the scores a selective index puts at risk, because a cluster the search skips is evidence the model never sees. Parity on short reasoning tests and losses on long retrieval tests is the signature a reader should expect from this design, and it is the signature the data shows.

Every figure in this post is a Prefrontal AI measurement. No external replication exists yet, so the speedups and the accuracy parity should both be treated as claims awaiting independent checks, not as settled results.

Training From Scratch With N-LSA

The second variant, N-LSA, trains with LSA from the start instead of distilling it into a finished model. In Prefrontal AI's test, N-LSA reached a validation loss of 1.085, identical to full attention and ahead of the sliding window, NSA, and PISA baselines at 1.153, 1.117, and 1.100. Training wall clock time improved 1.24x, and per layer training speed improved 1.6x.

The scope of that test needs emphasis. The model was a hybrid of roughly 0.5B parameters, evaluated up to 128k tokens. Matching full attention loss at that scale does not establish the same parity at frontier scale or at 10M token training lengths, and the release does not claim otherwise. What the result does establish is narrower and still useful. LSA is trainable from scratch without an obvious loss penalty at small scale, which keeps the from scratch path open while the distilled path carries the practical near term case.

Open Release Contents and Authorship

The release is unusually complete for a first publication. Prefrontal AI published the paper (arXiv:2610.42147, submitted October 30), the code with SGLang integration and context parallelism, H200 and B300 kernels, and training and inference details. SGLang integration is the detail that decides how quickly others can test the claims, because it places LSA inside a serving stack teams already run instead of leaving it as a standalone research artifact.

The author list includes Torsten Hoefler of ETH Zurich, a name associated with large scale systems work, and the team also proposed PaperTrail, an evaluation aimed at long context scientific tasks. Prefrontal AI describes this as one of its first open research releases. That youth cuts both ways. The artifacts are open enough for fast external scrutiny, and there is no track record yet by which to judge how the team's numbers usually hold up under that scrutiny.

Practical Meaning for Long Context Serving

For teams serving agent workloads, the distilled path is the one that can be tested this quarter. D-LSA requires an existing model, a short distillation step, and the published kernels, and it targets the exact cost that makes 10M token contexts impractical today. If the prefill figures reproduce even approximately, long context prefill stops being the binding constraint and the bottleneck moves to memory capacity and decode, a different and more familiar engineering problem.

The accuracy pattern argues for workload specific adoption rather than a blanket swap. Workloads dominated by reasoning over a provided context look well served by the reported numbers. Workloads that live or die on retrieving exact evidence from the middle of a very long context are the ones showing 5 to 8 point losses, and those teams should benchmark LSA on their own retrieval tests before trusting the headline parity. The release gives them everything needed to run that check, which is the strongest part of this announcement.

Related Reading

Attention cost is one side of inference performance. The hardware side of the same problem is covered in TorchTPU: Native PyTorch Backend for TPUs, 52% MFU Claimed, and in gpu-unlock for RTX 30: BF16 Gains up to 80% Claimed. Both posts deal with getting more work out of fixed silicon, the constraint LSA approaches from the algorithm side instead.

FAQ

What is Latent Sparse Attention?

Latent Sparse Attention is an attention method from Prefrontal AI that clusters keys hierarchically in latent space by meaning rather than by position. A query scores the top level clusters first and searches only inside the relevant ones, so fully hierarchical attention cost falls from quadratic O(T2) to near quadratic O(T log T).

How much faster is LSA at 10M tokens?

In Prefrontal AI measurements, D-LSA distilled into Qwen-3.8-27B made attention layer prefill 72.4 times faster than full attention on H200 at 10M tokens, and end to end serving 9.2 times faster. On B300 hardware, prefill was 42.9 times faster. All of these figures are self reported by Prefrontal AI.

Does LSA keep full attention accuracy?

On knowledge and reasoning benchmarks, both tested models stayed within 2 points of the originals, including MMLU-Pro at 83.2 to 83.0 and GPQA-D at 88.4 to 87.9. The parity is not uniform. Qwen fell 5 points on LongBench v2, from 70.0 to 65.0, and GLM lagged by up to about 8 points on RULER in the 2M to 5M token range.

What is the difference between D-LSA and N-LSA?

D-LSA is LSA distilled into an existing trained model with a short distillation step, which is the path behind the 72.4x prefill result. N-LSA is LSA trained from scratch. In a roughly 0.5B hybrid test model, N-LSA matched full attention validation loss at 1.085 while training 1.24 times faster in wall clock time.

Has anyone outside Prefrontal AI verified the results?

No. Every figure in the release comes from Prefrontal AI measurements, and no external replication has been published yet. The code, including SGLang integration and H200 and B300 kernels, is public, so independent checks are possible from the release artifacts.

Sources

PerformanceAI InfraAX Trend