Kenneth W. Bingham

research · side project · the method, in the open

Store the rule and the corrections, not the bytes.

This is the method behind geometry as a data source. A short rule predicts the data, and only the places where the prediction is wrong are stored. Rebuilding is exact every time. The whole method fits in one file you can run.

The problem

Storing a rule instead of data is an old idea, and it usually fails the same way. A rule never fits real data perfectly, so the leftover corrections end up almost as large as the original, and nothing is saved. Keeping those corrections small is the hard part.

The method, in four steps

Put simply, the rule paints the smooth parts, like paint by numbers, and the residual keeps the edges the rule cannot paint. The gain is roughly how much of the data is smooth fill compared with edges.

The proof

Each case is 256 by 256 bytes (65,536 raw). The baseline is the best existing lossless method tried on each one: brotli, deflate and zstd, each on the raw bytes and on PNG-style Up and Paeth filtered bytes. Every capsule rebuilt the original exactly.

What it does not claim

Run it yourself

One file, Node 18 or later, no packages: residual-method.js. Run node residual-method.js and it prints every number above, along with the limits.

14.7×smaller than best existing, field with a ripple
100%lossless, SHA-256 checked
<1×on random data, as it should be
1file, no packages