r/bcachefs

Hi, I find bcachefs very promising and have been following it for quite a while now. Since the "experimental" flag is gone, I'm seriously considering switching to bcachefs. So I am currently looking into whether it can support a snapshot workflow similar to the one I use currently.

I currently use Btrfs together with btrbk. One feature I find particularly useful is the onchange snapshot policy of btrbk.

With this policy, btrbk creates a new snapshot only when the source subvolume has changed since the previous snapshot. According to the btrbk documentation, this is implemented using the "Btrfs generation counter", which allows the tool to detect changes efficiently without comparing every file in the subvolume with the previous snapshot.

Hence the question: Does bcachefs happen to offer a similar mechanism that could be exploited for this purpose?

More specifically, is there some kind of generation ID, transaction ID, change counter, or other metadata that can be queried to determine whether a subvolume differs from an existent snapshot.

I am looking for an efficient check that does not require walking the entire directory tree or comparing all files between the current subvolume and the latest snapshot or relies on "inotifying" a whole subvolume.

My goal is to retain only snapshots whose filesystem state actually differs from the preceding snapshot to avoid multiple identical (hourly) snapshots.
Furthermore, I want to prevent the hard drives from spinning up unnecessarily just to create a snapshot that is identical to an already existing one. Could this second problem possibly be solved simply by pinning the metadata to the SSDs?

Hello. I got 1.5TB worth of files. It's a single encrypted SSD with full length MAC and zstd compression option

I started working on this recently because I found out that most if not all filesystem benchmarks still test the way it was done when ext2 was state of the art: one disk, default mkfs, raw throughput. But nobody picks btrfs, ZFS or bcachefs for throughput - you pick them for snapshots, redundancy, checksums, self-healing. Almost nobody publishes numbers on those.

So this suite benchmarks the machinery instead, across 17 configurations (btrfs/ZFS/bcachefs plus ext4/xfs over md/LVM as classic baselines, plus encryption variants): aging under 100 snapshots, corrupting a device behind the filesystem's back and checking whether scrub repairs it, failing a device and timing the rebuild, filling to hard ENOSPC, fsync tail latency, and "how long until my prompt comes back while a big cp runs in the background".

Yes, it runs on ephemeral GitHub runners with loop devices - I know what that means. Absolute MB/s is meaningless there, so the suite is built around shapes, ratios and verdicts, with per-VM calibration probes (lemon runners get auto-rerun) and conclusions drawn from trends over many runs. Real hardware with tiered topologies (NVMe cache over rotational disks, special vdevs) is the next step. I just need to gather hardware for it first.

What I'm really asking for is your eyes. Every number on the dashboard links to a description of exactly what ran, with what parameters, and to the code responsible - so if I picked a wrong mount option, an unfair default, a workload that misrepresents your favorite filesystem, or a broken measurement, you can find it and call it out. Several of the current tests exist because people poked holes in earlier results, and honestly that feedback improved this more than anything I did alone.

Dashboard: https://bartosz.fenski.pl/modern-fs-benchmark/
Repo: https://github.com/fenio/modern-fs-benchmark

NASty is a NAS operating system built on NixOS and bcachefs. It turns commodity hardware into a storage appliance serving NFS, SMB, iSCSI, and NVMe-oF - managed from a single web UI, updated atomically, and rolled back when things go sideways.

The highlights:

🪪 Active Directory — both directions

NASty can now join an existing AD domain as a member, or be the domain controller itself. As a DC you get integrated DNS and Kerberos, user / group / computer management from the web UI, on-box domain backups, and RSAT compatibility so the Windows admin tools work against it. (Experimental, single-DC for now.)

♻️ Backup restore + disaster recovery

Backups finally got their missing half. Restore a whole snapshot back onto your filesystem — including disaster recovery onto a bare box: point a fresh install at an existing backup repository, pick a snapshot, and pull it back.

🔥 Built-in firewall

An engine-managed nftables firewall, deny-by-default, with per-service source/interface restrictions and custom port rules for anything running outside NASty's service model (IPv4 + IPv6, TCP/UDP, port ranges).

⚡ RDMA over RoCE & InfiniBand

Optional zero-copy transports — iSER, NVMe-oF/RDMA and NFS-RDMA — for RoCE and InfiniBand NICs. The controls only appear when you actually have a capable card.

🖥️ SR-IOV & hardware passthrough

IOMMU-group view, vfio-pci toggles that survive reboots, per-device passthrough, and SR-IOV virtual-function management — VF count plus per-VF VLAN, MAC, trust and spoof-check.

📊 Operations control center

One page to see and drive every bcachefs array operation across your pools — start or cancel a scrub, watch a device evacuation, pause or resume reconcile and copy-GC — each row labeled with its pool.

…and a lot of smaller stuff

• Calculate folder sizes on demand right in the file browser
• .env files for Docker Compose stacks
• Remove leftover Docker networks from the UI (e.g. one an app brought with it)
• Secure Boot moved to lanzaboote v1.1.0 — validated on real UEFI + TPM2 hardware
• Linux 6.18.38 and bcachefs 1.38.8
• iSCSI portal management, VM disk/network editing, and a big pile of fixes + dependency refreshes

At my wits end trying to debug why the reconcile thread seems to do nothing on my pool.

[root@nasty:~]# bcachefs reconcile status /fs/add40cf7/
Scan pending:     0
                   data  metadata  
  replicas:           0         0  
  checksum:       7.34G         0  
  erasure_code:   12.0G         0  
  compression:    1.41M         0  
  target:             0         0  
  high_priority:      0         0  
  pending:        12.0G         0  
  stripes:            0         0  


processing normal phys: POS_MIN

Reconcile thread backtrace:
  [<0>] __bch2_closure_sync_timeout+0x3c/0x140 [bcachefs]
  [<0>] bch2_closure_sync_unbounded+0x2a/0x40 [bcachefs]
  [<0>] do_reconcile+0x86a/0xef0 [bcachefs]
  [<0>] bch2_reconcile_thread+0xfc/0x120 [bcachefs]
  [<0>] kthread+0xfb/0x260
  [<0>] ret_from_fork+0x22d/0x260
  [<0>] ret_from_fork_asm+0x1a/0x30

The checksum line has been stuck at 7.34G for weeks. I changed kernels and distributions in the mean time, but it won't advance. The erasure_code line is new; I only recently started a test with erasure coding, it's only enabled for a specific test subvolume, holding 3 files of 4GB each.

I've recently added a 3rd large HDD, but it doesn't even look like it's getting any user data.

All the while, the disks are spinning, and it seems that writes are happening:

https://preview.redd.it/q350y9oc8cdh1.png?width=664&format=png&auto=webp&s=aacd6d8184201d708655e05e439b3d90b89a589b

Superblock information: https://pastebin.com/uPFtvBBN

It could be possible that this may have started with some SATA controller issues, I've seen some `ATA: hard resetting link` messages in `dmesg` in the past, perhaps due to a disk not being properly seated; alas I haven't saved them.

However I don't understand why this whole drive activity happens constantly, and why the reconcile thread doesn't advance. Help is very much appreciated.

EDIT: When I stop the reconcile thread, most/all the disk activity stops.