u/PedulliF

Hi r/compression,

I’m sharing a narrow benchmark result for an experimental private lossless compressor and would like technical feedback / independent sanity checks.

This is not a global SOTA claim. It is only a measured Canterbury corpus result.

Benchmark:

Dataset: Canterbury corpus
Raw total size: 2,810,784 bytes
Round-trip decode: exact
All compressed artifact bytes counted: yes
Baseline: xz -9e

Results:

Method: Experimental private lossless compressor
Compressed size: 445,208 bytes
Exact round-trip: YES

Method: xz -9e
Compressed size: 493,080 bytes
Exact round-trip: YES

Main measured comparison:

445,208 < 493,080

So on this Canterbury run, the private compressor output is 47,872 bytes smaller than my measured xz -9e baseline.

Exact claim:

On my Canterbury corpus run, this experimental private lossless compressor produced a 445,208-byte artifact, decoded exactly back to the original corpus, and was smaller than my measured xz -9e baseline of 493,080 bytes.

I am not claiming that this beats xz universally, nor that it wins on every corpus. I am posting this to get benchmark criticism and reproducibility feedback.

Verification summary:

raw_total_bytes = 2,810,784
private_compressed = 445,208
xz_9e_compressed = 493,080
decode_exact = YES
sha256_match = YES

Round-trip verification method:

1. Hash original Canterbury input.
2. Compress with the private compressor.
3. Decompress the compressed artifact.
4. Hash decoded output.
5. Compare original and decoded output byte-for-byte.
6. Compare compressed artifact size against xz -9e.

Expected verification result:

SHA256 original == SHA256 decoded
byte-for-byte comparison returns success
compressed artifact size = 445,208 bytes

xz baseline command used:

xz -9e -k -c original_canterbury_input > canterbury.xz

Private compressor verification structure:

private_compressor compress original_canterbury_input output.private
private_compressor decompress output.private decoded_canterbury_output
cmp original_canterbury_input decoded_canterbury_output
wc -c output.private

Result:

output.private = 445,208 bytes
decoded output matches original exactly

Proof material:

I have a sanitized verification bundle containing the size logs, SHA256 checks, xz baseline log, and round-trip comparison log. I am keeping the implementation private for now to avoid leaking source code or algorithm details, but I can share sanitized verification material for audit/review.

What I’m asking for:

I’d appreciate feedback on whether the benchmark procedure is fair, whether xz -9e is a reasonable baseline here, what other baselines I should include, whether there is any hidden overhead I may be missing, and how best to package this for independent reproduction.

Again: this is a narrow Canterbury result, not a universal compression claim.

EDIT — fixed codec accounting:

A commenter correctly pointed out that codec/decompressor size should be disclosed.

In this setup the compressor/decompressor is the same fixed program used in encode/decode modes, so I count the fixed codec once, not twice.

Canterbury accounting:

• Private compressed output: 445,208 bytes
• Fixed codec as gzipped source: 9,780 bytes
• Output + gzipped codec source: 454,988 bytes
• Fixed codec as raw source: 36,590 bytes
• Output + raw codec source: 481,798 bytes
• Fixed codec as full unstripped executable: 78,884 bytes
• Output + full unstripped executable: 524,092 bytes
• xz -9e baseline: 493,080 bytes

So the Canterbury result remains under xz -9e when the fixed codec is counted as gzipped source or raw source.

Full disclosure: if I count the full unstripped executable binary instead, the total is 524,092 bytes, which is above xz -9e.

Corrected precise claim:

This is a bounded Canterbury win under source-count accounting, with byte-exact reconstruction.

It is not a universal compression claim, not a Hutter Prize claim, and not a multi-corpus/global claim. Silesia is measured but not yet under xz; Hutter is separate and should not be counted as a win unless final bytes say so.

Since the implementation is private, the fixed-codec-size claim would need independent verification under appropriate terms. I’m keeping the accounting public while avoiding source-code or algorithm disclosure.

---

Just to clarify the accounting philosophy:

My long-term intention is for the codec to be self-hosting / standalone, where the fixed codec representation can itself be represented through the same compression system. I understand that this is not customary benchmark accounting, and I do not want to use circular accounting as the main public claim.

So for the public Canterbury comparison, I’m using the conservative accounting:

• compressed output
• plus the fixed codec source counted once as raw source
• compared against xz -9e

That gives:

• Private compressed output: 445,208 bytes
• Fixed codec raw source: 36,590 bytes
• Output + raw source: 481,798 bytes
• xz -9e baseline: 493,080 bytes

So the clean claim is that Canterbury remains under xz -9e even with the fixed codec counted as raw source.

Separately, I may study self-hosted / internally compressed codec accounting, but I would treat that as an experimental/informational number, not the headline benchmark, unless the community agrees on a fair way to count it.