Subscribe
0%
Illustration for: Tufa Labs' tufazip squeezes 1 GB of Wikipedia into 105.39 MB with a transformer alone. What it shows, and its limits

Tufa Labs' tufazip squeezes 1 GB of Wikipedia into 105.39 MB with a transformer alone. What it shows, and its limits

Tufazip v2 is now listed on the Large Text Compression Benchmark at 105,388,353 bytes for enwik9, the best result there from a compressor whose only predictor is a neural network. It needs two Nvidia B200s and about a week each way, and its rank depends on which table you read.

Listen to this article0:00 / 5:22

Tufa Labs' tufazip has compressed enwik9, the standard 1 GB Wikipedia test file, down to 105,388,353 bytes without losing a bit. That's the total for version 2, including the decompressor, on Matt Mahoney's Large Text Compression Benchmark, which added the entry in an update dated November 19. Tufa Labs says it's the best result on the benchmark from a compressor that uses a neural network as its only predictor. The entries above it, by its account, combine a transformer with hand-built models.

Why it matters: compression and language modelling are the same problem. A model that predicts the next piece of text well can encode it in fewer bits. Tufazip is a clean test of that idea: no hand-tuned context mixing, just one transformer that learns while it compresses. It also shows what that costs.

How it works

According to the Tufa Labs write-up by Tommy He:

  • Model: a 12-layer, decoder-only transformer with 163M parameters (RoPE, SwiGLU, RMSNorm), a 4K word vocabulary and a 192-token context.
  • Online learning: the encoder and decoder each predict, code and update in lockstep. Because both sides learn the same weights from the same data, the weights learned along the way "never need to be transferred."
  • Warm start (v2): instead of starting from scratch, v2 starts from a quantized offline prior: a 34M-parameter model stored in 4.33 MB and expanded into the 12-layer network.
  • Replay: revisiting earlier text cut the payload from 107.22 MB to 105.92 MB in one experiment, at the cost of more compute on both sides.

The numbers

Size (bytes)
tufazip v2: compressed enwik9 100,958,324
tufazip v2: decompressor 4,430,029
tufazip v2: total 105,388,353
tufazip 0.1.0 (v1): total 105,979,069
DNA-4: total (next entry below) 105,554,360
nncp v3.2: total 107,261,318
cmix v21: total 108,244,767

v2 figures and neighbours: Large Text Compression Benchmark table. v1 figures: tufazip GitHub README.

The warm start saved 590,716 bytes over v1, as Glitchwire notes (our check of the two totals agrees). Glitchwire puts v2 at 0.843 bits per byte. Tufa Labs' post gives v1 as 0.848.

The cost: a week on two B200s

This isn't a compressor you'll run on a laptop. The GitHub repo requires Linux and exactly two Nvidia B200 GPUs (CUDA 13.0), and lists the test machine as an Nvidia DGX B200. Its submission table for v1 gives about 647,595 seconds to compress and 630,136 seconds to decompress, roughly 7.5 and 7.3 days, with peak memory of 68,525 MiB and 73,748 MiB respectively. The benchmark's v2 row lists the same two times and a 73,748 memory figure. Digit and Glitchwire give v2 slightly different timings (about 631,548 s and 645,748 s). Either way, it's about a week in each direction.

Where it actually ranks

The coverage disagrees here, so read headlines carefully:

  • Tufa Labs' post says it "took #2" on the benchmark, dated to the time of its September 7 submission.
  • Glitchwire says v2 "takes second place" behind fx2-cmix-transformer.
  • In the benchmark's main table as we read it, tufazip v2 is the 13th row, and fx2-cmix-transformer (96,996,198 bytes) isn't first. ex-girlfriend 2.0 leads at 88,983,124. explainx says 16th.

What holds up across sources is the narrower claim: it's the best entry whose sole predictor is a neural network. Digit says the same thing, noting that tufazip "cannot claim the top position."

Limits Tufa Labs names

The write-up says the system is tuned to one fixed file, and acknowledges a Goodhart problem: optimising for enwik9 isn't the same as building a general compressor. Bigger didn't help in the settings it tested, either. A 267M-parameter model did worse on enwik9 than a 188M one.

There's also a gap in what's public. The GitHub repo, MIT-licensed with a single commit when we checked, documents a deterministic decoder for the v1 artifact (105,924,360 bytes), not the v2 result.

Who should care

If you work on LLM efficiency, tufazip is a useful data point: one transformer, learning online, now beats long-standing neural compressors like NNCP on a fixed text. If you need practical compression, nothing here changes your tools. The week of B200 time to decode a gigabyte is the point, not a footnote. Watch for whether Tufa Labs publishes the v2 artifact and code, and whether independent reruns reproduce the 105,388,353-byte figure.

Sources

  1. Tufa Labs — tufazip: Learning to Compress a Billion Bytes of Wikipedia tufalabs.ai
  2. Matt Mahoney — Large Text Compression Benchmark mattmahoney.net
  3. GitHub — Tufalabs/tufazip github.com
  4. Digit — Tufazip explained: The transformer that squeezes 1GB of Wikipedia into 105MB digit.in
  5. Glitchwire — Tufazip Compresses 1 GB of Wikipedia to 105 MB Using Only a Transformer glitchwire.com
  6. explainx — tufazip: 1 GB of Wikipedia to 105.39 MB, Lossless explainx.ai

#Tufa Labs#tufazip#compression#transformers#enwik9#Large Text Compression Benchmark#Nvidia B200#open source