Btrfs/ZFS/bcachefs under workloads classic benchmarks skip

Interesting although I'd have liked more summaries: there's an awful lot there.

But the reasons I choose filesystems are more about reliability, failure modes, surrounding tooling, and so on.

Btrfs fails in several critical areas:

1. No way to accurately find free space

2. catastrophic failure on write if a volume fills up, the probability of which is greater because of #1

3. repair tools usually do not recover a corrupted volume and in my testing are most likely to render as damaged volume completely unreadable, which makes #2 worse

Put these things together and I can never trust Btrfs again. In the 9 years since I encountered these, I see no effort to fix them, just fooling around witg unimportant side details like performance tweaks.

Fix the critical issues first then make it faster.

The author of the benchmark here. I went over some comments and I'll try to tackle them here. I'm pretty clear that GH runner based benchmark is far from perfect due to noisy neighbours etc. Thus every test first is running so called calibration... to reject completely unreliable VMs. I'm fully aware that this can't completely fix the issue. Can limit it but not fix. But as of now there are 593 runs recorded so average should still be quite meaningful.

Having that said I'm desperately trying to get REAL hardware to run that benchmark. With some successes ;)

Few months ago I got Hetzner machine from Kent Overstreet and I was able to finish 3 runs before machine died... Results: https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/

Currently I've got even more interesting machine with tons of disks and I'm running new set of benchmarks but it's really in its initial stage.

https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/ 2nd run in progress... one run on REAL hardware takes much more time than on GH runner so it's slow.

But this new hardware has also so many disks that the plan is to try also more complex, tiered cache topologies. I'm working on it.

I'm happy to answer any other questions, sources of every piece of this benchmark are freely available and I'm not saying they are 100% correct. I'm open to improvements.

Seeing great results from bcachefs just makes me more sad that Kent and the other kernel devs couldn't come to an understanding to keep bcachefs in-tree. I want to use it for my storage arrays so badly, but I'm still stuck with btrfs as the only available in-tree filesystem with modern features.

> CI runs use loop devices on shared ephemeral VMs (one VM per filesystem): compare shapes and ratios, not absolute MB/s. Each job records a host-calibration anchor — see the table.

I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?

So two filesystems that are essentially shunned from the Linux kernel and permanent second-class citizens, and one that was removed from Red Hat and has a questionable history of reliability. Oh boy which do I choose?

I'm saying ZFS on another OS.

BCacheFS is the best Linux filesystem now that storage costs a premium.

You can mix devices of different sizes and types on bcachefs. You can have foreground and background devices to balance performance and also different compression settings for foreground and background transactions.

You can set replicas=N to the individual file or directory on bcachefs. For example files you can just re-download or re-build. Likewise you can set a higher number of copies to important files.

In case someone wonders why bcachefs is sadly no longer part of the mainline kernel. Start reading here: https://lwn.net/Articles/1027289/

First hybrid tier run: https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/hybrid... It is FIRST run... I will probably start tuning it now. And I'm open for suggestions what and how to tune ;)

What is md-raid10 doing that is so much worse than lvm-raid10? In terms of "I/O" and "responsiveness." It's not really obvious to me from either the linked page or https://github.com/fenio/modern-fs-benchmark . In principle they should be similar?

I'd be interested to see what the upcoming Btrfs performance improvements in linux 7.3 do for these benchmarks.

https://www.phoronix.com/review/linux-73-btrfs

Excuse my ignorance, but why didn't you benchmark on a RAM disk for more accurate results?

The numbers are quite interesting, does bcachefs run on Almalinux 10? I only see Fedora supported. All my servers run Almalinux today and I am super happy with it, no plans to use something else.

How is root support for bcachefs? This is the one thing I really miss with ZFS today, its just too much work. I do love my FreeBSD systems with a root ZFS though.

I think the reviews should also include the social aspect of these filesystems...

There is and have been many promising and exciting FS to replace the old boring ones, but for storage you not only want to avoid technical issues but also maintainer(s) drama...

> Every push/2-hourly cron builds each filesystem across 4 loop devices backed by sparse files, runs the suite, and publishes a results table in the job summary plus JSON artifacts.

I get that real hardware costs (author mentions EUR 70 a month for a suitable server), but without at least a baseline snapshot comparison run between real hardware, both SSD and HDD, and the sparse file-backed loop devices, it's hard to take much away from this.

Sadly the AI apocalypse isn't making stuff like this easy to do as a hobby.

Oh, so bcachefs is doing pretty well.