Context Compression Quality Benchmark: Compression Ratio vs Information Preservation

Token reduction is easy to inflate. Drop the one line the agent needed and the headline number goes up while the agent re-runs the command, retries in circles, or ships a worse answer. This benchmark measures both sides of that trade for sqz, on the exact code path the shell hook uses, and it runs as a regression test on every change.

Reproduce it:

cargo test -p sqz-cli --test quality_bench -- --nocapture

Source: sqz/tests/quality_bench.rs. The fixtures, facts and gates are all in that one file.

What is measured

Each fixture is realistic command output paired with a list of facts an agent would need from it: the failing test name, the assertion, the file:line, the changed lines of a diff, the pod in CrashLoopBackOff, the vulnerability count. The fixture is pushed through the real sqz binary exactly as the hook does (sqz compress --cmd <cmd> on stdin, isolated database via SQZ_DB_PATH), and five things are recorded:

Metric	How
Token reduction	`ceil(bytes / 4)` on input vs everything the agent sees: compressed stdout plus sqz’s own stderr lines. The `[sqz] N/M tokens` header is counted against sqz.
Critical-fact recall	Every fact must appear verbatim in the compressed stdout. One missed fact fails the test.
Noise dropped	Lines the fixture expects compression to remove (progress dots, deprecation chatter, passing tests). Reported, not gated.
Verbatim preservation	The `git show rev:path`, traceback and secrets fixtures must come out identical to the input, up to trailing whitespace.
Byte-exact recovery	A second run of the same output must return a `§ref:HASH§` dedup token, and `sqz expand HASH` must return the original bytes. Sensitive content must instead never be persisted.

Facts are chosen for what an agent needs to act, not for what sqz happens to keep. When the two disagreed during development, the formatter was fixed, not the fixture (see the bug list below).

Results

sqz main after 1.9.0, October 2026, default settings (dedup on, abbreviation on).

Fixture	Command	Tokens in	Tokens out	Reduction	Facts kept	Noise dropped	Recovery
grep, 45 matches in 3 files	`grep -rn compute_fee src tests`	667	479	28.2%	48/48	n/a	byte-exact
find, 80 files	`find . -name '*.py'`	515	515	0.0%	80/80	n/a	byte-exact
ls -la, 40 entries	`ls -la reports`	592	592	0.0%	40/40	n/a	byte-exact
kubectl get pods -o json	`kubectl get pods -o json`	1671	627	62.5%	13/13	n/a	byte-exact
kubectl apply with an error	`kubectl apply -f k8s/`	89	89	0.0%	3/3	0/1	byte-exact
aws ec2 describe-instances, 40	`aws ec2 describe-instances`	4693	1412	69.9%	80/80	n/a	byte-exact
gh pr view –json	`gh pr view 214 --json title,body,comments,files`	109	109	0.0%	4/4	n/a	byte-exact
gh pr diff	`gh pr diff 214`	294	205	30.3%	4/4	n/a	byte-exact
terraform plan	`terraform plan`	159	103	35.2%	3/3	1/1	byte-exact
npm install failure	`npm install`	153	153	0.0%	3/3	n/a	byte-exact
git push rejected	`git push`	145	92	36.6%	3/3	1/1	byte-exact
git show rev:path	`git show HEAD~3:src/billing/refund.py`	62	62	0.0%	2/2	n/a	byte-exact
cargo test failure	`cargo test`	303	154	49.2%	5/5	2/3	byte-exact
pytest failure	`pytest`	311	71	77.2%	5/5	3/3	byte-exact
tsc errors	`tsc --noEmit`	126	92	27.0%	6/6	2/2	byte-exact
git diff	`git diff`	148	148	0.0%	7/7	0/1	byte-exact
git status	`git status`	155	82	47.1%	6/6	3/3	byte-exact
npm install	`npm install`	161	39	75.8%	4/4	2/2	byte-exact
kubectl get pods	`kubectl get pods`	183	183	0.0%	10/10	n/a	byte-exact
docker ps	`docker ps`	193	79	59.1%	6/6	1/1	byte-exact
JSON API response	`curl https://api.example.com/v1/orders/ord_8812`	129	130	-0.8%	11/11	1/1	byte-exact
app log with repeats	`tail -n 200 app.log`	1025	135	86.8%	3/3	2/2	byte-exact
source file read	`cat src/billing/refund.py`	221	221	0.0%	5/5	n/a	byte-exact
python traceback	`python app.py`	82	82	0.0%	4/4	n/a	passthrough, never cached
dotenv with secrets	`cat .env`	78	78	0.0%	4/4	n/a	passthrough, never cached
Aggregate		12,264	5,932	51.6%	359/359

Regression gates enforced by the test: 100% fact recall on every fixture, output identical to the input (up to trailing whitespace) for the git show rev:path, traceback and secrets fixtures, byte-exact recovery for everything else, no persistence of the traceback and secrets fixtures, and at least 40% aggregate reduction across the other fixtures.

Reading the zeros

The 0.0% rows are deliberate and are the point of the benchmark. In each of them sqz returned the input unchanged: either nothing was safe to remove, or what could be removed saved fewer than 16 tokens, and the net-win gate passes the original through rather than add a header for that.

find, ls -la. No per-command formatter; both go through the generic pipeline. Output where most lines are records (bare paths, ls -l entries) is never entropy-truncated below 16,000 characters, so every line stays, and collapsing the ls -la column padding saves almost nothing in real tokens.
kubectl get pods, gh pr view –json. Their formatters were removed because they replaced the data with a summary. The generic pipeline keeps every row and field, and the table padding and the small JSON object leave less than 16 tokens to save.
kubectl apply with an error, npm install failure. A formatter’s result is not used when it would lose a line that reports the failure (Error from server, npm error). The output then goes through the generic pipeline, which keeps error lines and here found less than 16 tokens to save.
git show rev:path. The git show formatter passes file contents through, and the pipeline treats them as source code, like the source file read below.
git diff. Every changed line is a fact. sqz’s diff formatter found nothing safe to remove, and the generic pipeline agreed, so the output passed through untouched with no stats header. An earlier build reported this same fixture as a 15% reduction because it compared a BPE token count for the input against a bytes/4 estimate for the output. That mismatch inflated every pipeline-path number in sqz stats and is fixed.
source file read. An agent that runs cat on a file is usually about to edit it. The pipeline now detects source code and skips every line-dropping stage. Before this, the entropy truncator scored the imports and the class line as “low information” and removed them. The saving on file reads comes from dedup instead: the second read is 13 tokens.
python traceback. The Traceback (most recent call last): line followed by File lines routes it to safe mode, which passes it through unchanged at any size, a long chained traceback included. Like the dotenv fixture it is never written to the cache.
dotenv. The key assignments route it to safe mode, and credential lines pass through unchanged at any size. Safe-mode content is also never written to the cache on this path, which the test proves by checking that a repeat run does not produce a dedup reference.

The JSON API row is the one where the agent sees more than the raw output. Null stripping, flattening and TOON encoding cut the payload from 165 to 138 cl100k tokens, enough to pass the net-win gate, but the [sqz] header line, which repeats the full curl URL, costs more than that saving, so the total in this table comes out one token larger.

Regret: the metric that catches over-compression in the field

A fixture can only test facts someone thought of. For everything else, sqz tracks two signals in normal use and prints them in sqz stats:

Quick re-runs. The agent produced byte-identical output within 120 seconds of last seeing it (SQZ_REGRET_WINDOW_SECS to tune, 0 to disable). The repeat bought no new information, which usually means the compressed first serving was missing something. Grouped by command, so the formatter at fault is visible.
Ref expands. A §ref:HASH§ token was expanded back to its original bytes. A recovery round trip.

These are reported next to the savings total as a rate, not subtracted from it. The token cost of the re-run itself is small and misleading: the second serving is identical, so it hits the dedup cache and goes out as a 13-token reference. The real cost is the wasted agent turn, which sqz cannot see from tool output alone. Any dollar figure for it would be an invented multiplier, so sqz stats --cost models direct token cost only and says so.

What this benchmark does not measure

Agent task success. Whether a model given the compressed output completes the task as often as with the raw output. This needs a live model, a task set and many runs, and it is the number that matters most. Fact recall is the proxy used here because it is deterministic and cheap enough to run on every commit.
Real tokenizer counts. The table uses bytes/4; the [sqz] N/M tokens header counts cl100k tokens. cargo test -p sqz-engine cmd_formatter_bench re-measures the formatter fixtures with the cl100k BPE tokenizer and fails if chars/4 overstates the cl100k reduction by 10 points or more (latest run: 63.7% by chars/4, 74.0% by cl100k).
Session-level effects. Dedup across a whole session (the 13-token repeat) is where most real savings come from. That is measured separately in benchmark.md and in the sqz stats figures on the README, which come from real usage, not fixtures.
Abbreviation losses. N-gram abbreviation replaces repeated phrases with «A1» symbols plus a legend. A fact inside a repeated phrase still counts as kept if it appears in the legend. Use --no-abbrev or SQZ_NO_ABBREV=1 if an agent struggles with the symbols.

Bugs this benchmark found on its first run

Listed because they are the kind of loss that a ratio-only benchmark hides, and because they are now regression-tested.

Command	What was lost	Fix
`npm install`	The `3 vulnerabilities (1 moderate, 2 high)` line and the `npm audit fix` hint. An early `return` on the “added N packages” line skipped everything after it.	Collect all summary lines.
`docker ps`	Columns mis-parsed whenever COMMAND or CREATED contained spaces (`"uvicorn app:app"`, `2 hours ago`), so status and ports landed in the wrong cells.	Slice rows at the header’s column offsets instead of splitting on whitespace.
`git status`	The branch name and ahead/behind line.	Keep the leading branch and tracking lines; drop only “up to date”.
`pytest`	The `file:line` of the failure, the failing source line and the `1 failed, 41 passed` totals. Only the short-summary line survived.	Dedicated pytest formatter that keeps totals, `>` and `E` lines and the location.
`tsc`	Everything, when tsc used its `--pretty` format: the formatter only knew the piped `file(line,col)` layout, so output fell through to the generic pipeline, which dropped `Found 2 errors`.	Parse both formats, keep tsc’s own summary line.
`cat file.py`	Imports and the `class` declaration, scored as low-information by the entropy truncator.	Source-code detection disables line-dropping stages.
any passthrough	`sqz stats` recorded savings on output that was returned unchanged, because input and output were counted with different estimators.	Count both with the same tokenizer; apply the 16-token net-win gate on the formatter path too.

Adding a fixture

Add a fx(...) entry to fixtures() in sqz/tests/quality_bench.rs with the raw output, the facts an agent would need, and optional noise lines. If a fact goes missing, the test fails with the fixture name and the missing strings. If you use a command daily and sqz mangles it, that fixture plus the failing test is the best issue report you can file.

Keyboard shortcuts

sqz: context compression for AI coding agents