The kernel may write there at any time

macOS has had transparent file compression since 2011. It is called decmpfs, it is what makes a fresh install of the OS smaller than the sum of its files, and you can turn it on yourself with afsctool or applesauce.

It has one property that is rarely mentioned: compression does not survive a rewrite. Compress a file, open it, save it, and it is stored uncompressed again. For a photo archive that does not matter. For anything a program writes to — a build cache, a database, a VM image, a log — it means the compression you applied on Monday is gone by Tuesday.

I wanted the compression to live in the file system instead of in a file attribute, so that a rewritten block comes back compressed. macOS 15.4 introduced FSKit, which lets you write a file system in user space without a kernel extension. So I wrote one.

This is what it cost me.

The bug that ate other people's data

Somewhere around the fourth month I had a file system that passed its tests, copied 14,885 files at 580 files a second, and produced a checksum mismatch roughly once every three runs.

Once every three runs is the worst possible failure rate. It is frequent enough to be real and rare enough that a single clean run feels like a fix. Twice I “fixed” it and twice I was wrong — first by blaming hand-written @objc selectors, then by blaming early dictionary training. Both theories died the same way: the corruption reproduced on a build that had neither.

What I had not done was read the header.

FSKit has a call named blockmapFile. Your file system hands the kernel a map of physical extents, and the kernel does the I/O itself — no round trip through your process. That is where the speed comes from. The documentation comment on that call says the kernel keeps the mapping and “may perform I/O to this range (or a part of it) at any time” (FSKit, FSVolumeExtent.h).

At any time. Not “until you acknowledge the write”. Not “before completeIO” — which, incidentally, can arrive asynchronously and tell you nothing about whether the kernel is done with the map.

I was treating those extents as free the moment the write completed. When a file was truncated, its tail went straight back into the free list, got handed to the next file that asked for space, and then the kernel flushed a deferred write through the map it still held — on top of somebody else's data.

There were four places doing this: trimming tails past EOF, rewriting a block in place, releasing the old extent after a flush, and moving blocks during background recompression. All four were correct under the model in my head and wrong under the model in the header.

The fix is bookkeeping, not cleverness. Track which nodes have a live kernel mapping. When an extent is freed while a mapping is live, it goes to quarantine instead of the free list. Quarantine drains when the vnode is reclaimed and the mapping provably no longer exists — and even then into the pending list, not into immediate reuse, because of those asynchronous completions.

This is not an Apple bug. The behaviour is documented, in the file I should have read first. It is the kind of contract that is invisible until you violate it, and then it is invisible for a while longer, because it corrupts data belonging to a different file than the one you were touching.

Why the shipped build is not the build in this story

The bug above was found and fixed in development. But “I fixed it” is a weak thing to say to someone deciding whether to hand you their files, so here is what the format does instead of resting on my confidence.

Every block carries an XXH64 checksum, verified on read. Checksums do not prevent corruption, but they ensure corrupted data is never returned silently: a mismatch surfaces as an error naming the file and the block index.

Two superblocks with generation numbers. A commit writes the new snapshot into free space and only then switches the superblock, so a crash at any step leaves the previous consistent state intact. Commits never write over extents the last snapshot still references.

Clean shutdown is recorded. If a volume was detached uncleanly, the next mount says so and loads the last consistent state instead of guessing.

There is an integrity check that walks the superblock, the snapshot and every data block — in the app, and as cdrive check.

None of that turns a young file system into a proven one, and I would not claim otherwise. What it rules out is the failure that actually costs you data: the one nobody notices until the backups have rotated.

The methodology lesson is the bigger one

At a 25–50% failure rate, one clean run means nothing. It means nothing twice.

After the fix I ran eight volumes back to back in a single extension process: zero mismatches. On the version before, the same eight gave three mismatches, then three again. On the intermediate version, two. That is the only kind of evidence that was worth anything, and I did not start collecting it until after I had wasted two days on wrong theories built from single runs.

If your bug is probabilistic, your test has to be too. I now refuse to believe any storage fix that has not survived at least four volumes in a row.

Three other things macOS taught me

APFS materialises small sparse files. I packaged the container as a sparse bundle — a directory of fixed-size “band” files, the format Apple uses for sparse disk images. With Apple's default band size of 8 MiB, a volume holding a couple of hundred megabytes of compressed data occupied gigabytes on disk: every band that was touched at all had been allocated in full, hole and all.

There is a size below which APFS stops keeping the hole and just writes the whole extent out. I am deliberately not quoting the number, because when I went back to re-measure it before publishing this, I did not get the figure I had written down — and a threshold that moves between machines or OS versions is worth less than the method. The method is three lines:

python3 -c 'open("b","wb").truncate(32<<20)'
python3 -c 'f=open("b","r+b"); f.seek(16<<20); f.write(b"\0"*4096)'
stat -f '%z logical, %b blocks' b

Run that for a few band sizes on your own machine and you will see where the cliff is on yours. Mine is well above 8 MiB, which is all I needed to know: I use 32 MiB bands.

diskutil is not the disk arbitration layer. For a long time I believed FSKit volumes could not be unmounted programmatically, because diskutil unmount answered “failed to unmount” every time. The layer that refuses is storagekitd, sitting above DiskArbitration. A direct DADiskUnmount unmounts the volume without complaint, needs no root, and spawns no external process — which also happens to be the only way to do it from inside an App Store sandbox. I lost days to a diagnostic tool's opinion.

Paths arrive in two normalisations. A volume named Сжатый диск is stored decomposed on disk — the й is и plus a combining breve. The same path typed into a save dialog can stay composed. Compare them directly and they are not equal, silently, and a mounted volume appears disconnected. I found this by writing the fix wrong: I normalised one side, which broke exactly the case it was meant to protect. Normalise both sides or neither.

What the numbers actually look like

Compression ratios in marketing copy are chosen by the marketer. Here are mine, measured, including the one that is bad for me:

That last row is the honest answer to “should I use this”. If your disk is full of video, photos, music and games, it is already compressed, and a compressing file system will give you back 645 MB out of 100 GB while burning every core for five minutes. Do not buy it. If your disk is full of caches, dependency trees, dumps, logs and VM images, the first two rows are what you should expect.

Throughput is 580 files a second on the compact write path, against 588 with the kernel-offloaded path — a 1.4% difference that I expected to be much larger, and which is the reason the offloaded path is no longer the default.

Would I do it again

FSKit is young. The documentation exists but is thin, the contracts are load bearing, and a mistake costs you somebody's files rather than a stack trace. Every macOS update is a chance for the module to break, so this is maintenance, not a finished thing.

But writing a file system in Swift, in user space, that the Finder mounts on a double click, would have been a kernel extension and a code-signing saga two years ago. That part genuinely works.

What DenseDrive is

A DenseDrive disk is stored as a sparse bundle — a package that Finder shows as a single item and that holds fixed-size band files inside — kept in a folder you choose. Open it and it mounts in Finder as an ordinary volume: Spotlight indexes it, drag and drop works, any app can write to it without knowing what it is. Eject it and it collapses back to that one package, which you can copy to another Mac and open there.

It is worth pointing at caches, dependency trees, database dumps, logs and VM images. It is not worth pointing at a photo library. Volumes can optionally be encrypted with a password — AES-256-GCM, key derived with PBKDF2 — which the decmpfs-based compression tools do not offer. That protects the container at rest; it has had no independent security audit, and I would not present it as a security product.