The problem
Storing a rule instead of data is an old idea, and it usually fails the same way. A rule never fits
real data perfectly, so the leftover corrections end up almost as large as the original, and nothing
is saved. Keeping those corrections small is the hard part.
The method, in four steps
- Predict. A short rule builds the whole field from a few numbers. The examples use the
saddle z = x × y and a slice of the Schwarz D surface.
- Correct. The residual is the exact difference between the data and the prediction, byte by
byte. Prediction plus residual gives back the original, checked with SHA-256, so it is lossless by
construction.
- Pass it to the neighbour. Each residual byte is stored as its difference from the byte above
it. Where the rule predicts well, the residual is nearly all zeros, and what is left gathers along
edges and lines. Differences between neighbours along a smooth stretch are smaller still. This is
the same idea as PNG's Up filter, so it is not new on its own.
- Pack. A standard compressor packs what is left. The stored capsule is the rule plus the
packed residual.
Put simply, the rule paints the smooth parts, like paint by numbers, and the residual keeps the edges
the rule cannot paint. The gain is roughly how much of the data is smooth fill compared with edges.
The proof
Each case is 256 by 256 bytes (65,536 raw). The baseline is the best existing lossless method tried on
each one: brotli, deflate and zstd, each on the raw bytes and on PNG-style Up and Paeth filtered bytes.
Every capsule rebuilt the original exactly.
- Saddle field: best existing 2,476 bytes (brotli with the Up filter), capsule 44 bytes.
- Schwarz D slice: best existing 10,568 bytes, capsule 53 bytes.
- Saddle plus a ripple the rule does not know: best existing 2,618 bytes, capsule 178 bytes,
14.7 times smaller. This is the case that matters, because the rule does not describe everything.
- Random bytes: best existing 65,540 bytes, capsule 65,571 bytes. The method loses slightly,
as it should, and the existing method is the right choice.
What it does not claim
- The test fields are generated by the same rules that rebuild them, so they are the best case by
design. The first two results mostly show how small a formula is compared with its output.
- Each rule here is chosen by hand. Finding the right rule for unfamiliar data automatically is not
shown.
- On real photographs, PNG beat every scheme I tested. Real data has to be measured type by type
against the best method for that type, and geometric storage only belongs where it clearly wins.
- It saves storage. It is not faster than reading bytes that are already local.
Run it yourself
One file, Node 18 or later, no packages:
residual-method.js. Run
node residual-method.js and it prints every number above, along with the limits.
14.7×smaller than best existing, field with a ripple
100%lossless, SHA-256 checked
<1×on random data, as it should be
1file, no packages