CompressionBench Results: Coding Agents Compress LLMs
CompressionBench, a benchmark from a research team led by Xinrui Zhong and published on arXiv on November 4, 2026 (arXiv 2611.06858), asks whether coding agents can compress a large language model without a human driving the work. The answer, measured across six agents and six model and size settings, is mostly yes: 29 of 36 submissions passed independent verification. Picking the lowest perplexity agent design per task kept the average perplexity increase to 3.4% over the BF16 original, with downstream accuracy at 77.33% against 79.56% for BF16.
The framing matters as much as the score. The paper opens with recursive self-improvement, the idea that an AI system should be able to improve the models and infrastructure that later systems run on. That claim is usually argued in the abstract. CompressionBench turns one piece of it into a task with a hard budget and a checkable output: keep a pretrained model behaving like itself while fitting it into less storage. Compression is also practical work. Every on-device deployment needs it, and today it is still a job a specialist repeats by hand for each model and each target device.
Task Design and Agent Freedom
Each task arrives in terminal form. The agent receives a source checkpoint, a storage limit, an inference environment and a time budget. What it hands back is not a report but an artifact: compressed weights plus executable inference code that loads and runs them. The agent controls its own algorithm and its own experiments, which makes this a design task rather than an instruction following task. Two agents given the same checkpoint can reach the storage budget through different representation choices, and the benchmark scores the outcome of those choices.
That freedom is the point of the exercise. A fixed compression recipe applied by a script would test the recipe. CompressionBench tests judgement under a budget: which structure to keep, which precision to spend where, and when an experiment says a direction is failing. The paper names representation design, experimental judgement and capability validation as the three skills the results point to, and the task design is built so that all three show up in the trace.
Headline Results Against BF16
The pass rate is the first number to read. Seven of 36 submissions failed independent verification, so autonomous compression is not yet a routine the field can take for granted. The quality numbers for the passing designs are close to the original: a 3.4% average perplexity increase and a downstream accuracy gap of 2.23 points. Read together, the results say agents can already do this work at a level near the uncompressed baseline, and that the remaining distance sits in reliability, in the one submission in five that does not survive verification, rather than in a large quality collapse among the ones that do.
One caution belongs with the headline. The 3.4% and 77.33% figures come from selecting the lowest perplexity design per task after the fact. That is a best-of selection across agents, not the expected result of handing a checkpoint to one agent once. A team budgeting for agent driven compression should plan around the pass rate and the spread across agents, both of which the abstract reports only in aggregate. My read is that the benchmark proves capability at the frontier of the runs it collected, while the operating number for a single run is still an open question the full paper needs to answer.
Perplexity Feedback and Its Limits
The experiment traces are where the paper gets specific about how agents improve. At nearly fixed storage, agents lowered perplexity in 21 of 24 revisions. Perplexity feedback, in other words, is a usable compass for local search: change the design, read the score, keep what helps. A score that responds this reliably is what makes an autonomous loop possible at all, because the agent can iterate without waiting for a human to judge each step.
The same traces show where that compass misleads. Complete model feedback reversed 17 of 79 prior local rankings, a reminder that a design that wins on a local slice can lose when the whole model is evaluated. Generation and inference tests then exposed failures that likelihood alone misses. The practical lesson generalizes beyond compression: an agent optimizing a single fast metric will find the metric's blind spots, so the verification layer has to test behavior the metric does not cover. CompressionBench builds that layer in as an independent check, and the failures it caught are evidence the layer is doing real work.
The benchmark's most transferable finding is not the pass rate. It is the split between a local score that guides search well and a full check that still overturns local rankings and catches failures the score cannot see.
Implications for On-Device Deployment
Compression sits on the critical path of on-device AI. A model that does not fit the storage and memory budget of a phone or an embedded device does not ship, and the fitting work is currently manual, model by model and device by device. A benchmark where agents complete that work end to end, including the inference code, points at a pipeline where a new model release triggers agent compression runs against a list of target budgets, with verification as the gate before a human reviews the result.
The limits of the evidence should stay in view. The abstract does not name the six agents or the target models, so no agent ranking can be drawn from it. Per task time budgets and costs are not stated in the abstract, which means the economics of replacing a compression engineer with agent compute cannot be judged yet. Whether code is released is also unconfirmed from the abstract alone. Those gaps decide how directly teams can reproduce or adopt the method, and they are the first things to check in the full paper.
Open Questions and Unconfirmed Details
- Agent and model identities are unnamed in the abstract. Six agents and six model and size settings were used, but the abstract does not say which ones, so per agent comparisons are unconfirmed.
- Time budgets and costs are unstated. Each task carries a time budget, but its size and the compute cost per submission are not available from the abstract.
- Code release is unconfirmed. The abstract does not state whether the benchmark or the agent designs are publicly released.
- Best-per-task selection flatters the headline. The 3.4% perplexity increase and 77.33% accuracy select the best design per task, so single run expectations remain unmeasured in the abstract.
None of that weakens the core result. A coding agent's job description now has a measured extension beyond writing code: improving the efficiency of the model itself, inside a storage budget, with its output checked by an independent verifier. CompressionBench gives that extension numbers, a 29 of 36 pass rate, near baseline quality for the best designs, and a clear account of where the guiding metric stops being trustworthy.
Related Reading
For another benchmark that stresses a narrow capability until frontier systems break, see the report on StateBench and long document state tracking. For coding agents moving into everyday tooling, see Open WebUI v0.13.1 and its coding agent picker.
FAQ
What is CompressionBench?
CompressionBench is a benchmark from a team led by Xinrui Zhong that tests whether coding agents can compress LLMs autonomously. Each task is a terminal task: the agent receives a source checkpoint, a storage limit, an inference environment and a time budget, then builds compressed weights and executable inference code.
How many CompressionBench submissions passed verification?
29 of 36 submissions passed independent verification. The runs covered six agents across six model and size settings.
How close did agent compression get to the BF16 original?
Selecting the lowest perplexity agent design per task gave an average perplexity increase of 3.4% over BF16. Downstream accuracy was 77.33%, compared with 79.56% for BF16.
What did CompressionBench reveal about perplexity as a guide?
Perplexity feedback helped agents improve at nearly fixed storage, with 21 of 24 revisions lowering perplexity. But complete model feedback reversed 17 of 79 prior local rankings, and generation and inference tests exposed failures that likelihood alone misses.
What does CompressionBench leave unconfirmed?
The abstract does not name the six agents or the target models. The per task time budgets and costs, and whether code is released, are also not confirmed from the abstract alone.