CompressionBench Results: Coding Agents Compress LLMs

Agentic Benchmark · ax-trend issue #3831 · arXiv paper published November 4, 2026

CompressionBench title card

CompressionBench, a benchmark from a research team led by Xinrui Zhong and published on arXiv on November 4, 2026 (arXiv 2611.06858), asks whether coding agents can compress a large language model without a human driving the work. The answer, measured across six agents and six model and size settings, is mostly yes: 29 of 36 submissions passed independent verification. Picking the lowest perplexity agent design per task kept the average perplexity increase to 3.4% over the BF16 original, with downstream accuracy at 77.33% against 79.56% for BF16.

The framing matters as much as the score. The paper opens with recursive self-improvement, the idea that an AI system should be able to improve the models and infrastructure that later systems run on. That claim is usually argued in the abstract. CompressionBench turns one piece of it into a task with a hard budget and a checkable output: keep a pretrained model behaving like itself while fitting it into less storage. Compression is also practical work. Every on-device deployment needs it, and today it is still a job a specialist repeats by hand for each model and each target device.

Submissions passing independent verification29 of 36
Average perplexity increase over BF16, best design per task+3.4%
Downstream accuracy, best agent design per task77.33%
Downstream accuracy, BF16 original79.56%

Task Design and Agent Freedom

Each task arrives in terminal form. The agent receives a source checkpoint, a storage limit, an inference environment and a time budget. What it hands back is not a report but an artifact: compressed weights plus executable inference code that loads and runs them. The agent controls its own algorithm and its own experiments, which makes this a design task rather than an instruction following task. Two agents given the same checkpoint can reach the storage budget through different representation choices, and the benchmark scores the outcome of those choices.

That freedom is the point of the exercise. A fixed compression recipe applied by a script would test the recipe. CompressionBench tests judgement under a budget: which structure to keep, which precision to spend where, and when an experiment says a direction is failing. The paper names representation design, experimental judgement and capability validation as the three skills the results point to, and the task design is built so that all three show up in the trace.

Diagram of a CompressionBench task from checkpoint through agent compression and perplexity feedback to verification
The task flow: a checkpoint and a storage budget in, a verified compressed model out.

Headline Results Against BF16

Bar chart of downstream accuracy for BF16 at 79.56 percent and the best agent design per task at 77.33 percent
Downstream accuracy on an absolute 0 to 100 scale, best agent design per task against the BF16 original.

The pass rate is the first number to read. Seven of 36 submissions failed independent verification, so autonomous compression is not yet a routine the field can take for granted. The quality numbers for the passing designs are close to the original: a 3.4% average perplexity increase and a downstream accuracy gap of 2.23 points. Read together, the results say agents can already do this work at a level near the uncompressed baseline, and that the remaining distance sits in reliability, in the one submission in five that does not survive verification, rather than in a large quality collapse among the ones that do.

One caution belongs with the headline. The 3.4% and 77.33% figures come from selecting the lowest perplexity design per task after the fact. That is a best-of selection across agents, not the expected result of handing a checkpoint to one agent once. A team budgeting for agent driven compression should plan around the pass rate and the spread across agents, both of which the abstract reports only in aggregate. My read is that the benchmark proves capability at the frontier of the runs it collected, while the operating number for a single run is still an open question the full paper needs to answer.

Perplexity Feedback and Its Limits

The experiment traces are where the paper gets specific about how agents improve. At nearly fixed storage, agents lowered perplexity in 21 of 24 revisions. Perplexity feedback, in other words, is a usable compass for local search: change the design, read the score, keep what helps. A score that responds this reliably is what makes an autonomous loop possible at all, because the agent can iterate without waiting for a human to judge each step.

The same traces show where that compass misleads. Complete model feedback reversed 17 of 79 prior local rankings, a reminder that a design that wins on a local slice can lose when the whole model is evaluated. Generation and inference tests then exposed failures that likelihood alone misses. The practical lesson generalizes beyond compression: an agent optimizing a single fast metric will find the metric's blind spots, so the verification layer has to test behavior the metric does not cover. CompressionBench builds that layer in as an independent check, and the failures it caught are evidence the layer is doing real work.

The benchmark's most transferable finding is not the pass rate. It is the split between a local score that guides search well and a full check that still overturns local rankings and catches failures the score cannot see.

Implications for On-Device Deployment

Compression sits on the critical path of on-device AI. A model that does not fit the storage and memory budget of a phone or an embedded device does not ship, and the fitting work is currently manual, model by model and device by device. A benchmark where agents complete that work end to end, including the inference code, points at a pipeline where a new model release triggers agent compression runs against a list of target budgets, with verification as the gate before a human reviews the result.

The limits of the evidence should stay in view. The abstract does not name the six agents or the target models, so no agent ranking can be drawn from it. Per task time budgets and costs are not stated in the abstract, which means the economics of replacing a compression engineer with agent compute cannot be judged yet. Whether code is released is also unconfirmed from the abstract alone. Those gaps decide how directly teams can reproduce or adopt the method, and they are the first things to check in the full paper.

Open Questions and Unconfirmed Details

None of that weakens the core result. A coding agent's job description now has a measured extension beyond writing code: improving the efficiency of the model itself, inside a storage budget, with its output checked by an independent verifier. CompressionBench gives that extension numbers, a 29 of 36 pass rate, near baseline quality for the best designs, and a clear account of where the guiding metric stops being trustworthy.

Related Reading

For another benchmark that stresses a narrow capability until frontier systems break, see the report on StateBench and long document state tracking. For coding agents moving into everyday tooling, see Open WebUI v0.13.1 and its coding agent picker.

FAQ

What is CompressionBench?

CompressionBench is a benchmark from a team led by Xinrui Zhong that tests whether coding agents can compress LLMs autonomously. Each task is a terminal task: the agent receives a source checkpoint, a storage limit, an inference environment and a time budget, then builds compressed weights and executable inference code.

How many CompressionBench submissions passed verification?

29 of 36 submissions passed independent verification. The runs covered six agents across six model and size settings.

How close did agent compression get to the BF16 original?

Selecting the lowest perplexity agent design per task gave an average perplexity increase of 3.4% over BF16. Downstream accuracy was 77.33%, compared with 79.56% for BF16.

What did CompressionBench reveal about perplexity as a guide?

Perplexity feedback helped agents improve at nearly fixed storage, with 21 of 24 revisions lowering perplexity. But complete model feedback reversed 17 of 79 prior local rankings, and generation and inference tests exposed failures that likelihood alone misses.

What does CompressionBench leave unconfirmed?

The abstract does not name the six agents or the target models. The per task time budgets and costs, and whether code is released, are also not confirmed from the abstract alone.

Sources

Agentic BenchmarkAI CodingOn-DevicePerformanceAX Trend