Struct CompressionPipeline 

pub struct CompressionPipeline { /* private fields */ }

The compression pipeline orchestrator.

Runs content through a sequence of configurable stages (condense, strip nulls, collapse arrays, etc.), then applies post-processing: RLE compression, sliding-window dedup, entropy truncation, token pruning, dictionary compression, and TOON encoding for JSON.

Stages are sorted by priority — lower numbers run first. You can insert custom stages with insert_stage.

Implementations

impl CompressionPipeline

pub fn new(_preset: &Preset) -> Self

Construct the pipeline from a preset, creating all built-in stages sorted by priority, plus the token pruner, dictionary compressor, and entropy truncator.

pub fn compress( &self, input: &str, ctx: &SessionContext, preset: &Preset, ) -> Result<CompressedContent>

Run content through all enabled stages, then apply:

1. Dictionary compression + TOON encoding for JSON
2. Token pruning for prose/plain text
3. Entropy-weighted truncation for long content

pub fn compress_lossless( &self, input: &str, ctx: &SessionContext, preset: &Preset, ) -> Result<CompressedContent>

Byte-faithful compression for file reads (sqz_read_file, sqz_grep, sqz_list_dir). Returns the input verbatim apart from ANSI-escape stripping: every content-altering stage is skipped — both the droppers (entropy truncation, token pruning, JSON field projection, and the lossy configured stages: condense, strip_fields, strip_nulls, flatten, truncate_strings, collapse_arrays, git_diff_fold) and the reversible-but-reformatting ones (RLE, sliding-window dedup, TOON/dict), which would shift bytes or line numbers. No line, field, or segment is ever dropped or moved. Token savings on this path come from the dedup §ref cache on repeat reads, not from re-encoding. See https://github.com/ojuschugh1/sqz/issues/32

pub fn count_tokens(&self, text: &str, preset: &Preset) -> u32

Token count with the same tokenizer compress used for tokens_original, so callers that post-process the output can report comparable numbers.

pub fn insert_stage(&mut self, stage: Box<dyn CompressionStage>)

Insert a plugin stage and re-sort by priority.

pub fn reload_preset(&mut self, _preset: &Preset) -> Result<()>

Rebuild stage list from a new preset (hot-reload support). Built-in stages are recreated; plugin stages are dropped and must be re-inserted by the caller.