
MeshXFS: A clustered filesystem built on XFS, written by AI
A POSIX clustered filesystem built on XFS
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
A POSIX clustered filesystem built on XFS
Mount the same shared block device read-write from multiple nodes.
Shared SAN or DRBD storage · Network or on-disk locking · Fencing and journal recovery
MeshXFS is an open-source (GPL-2.0) clustered filesystem. Built on XFS, it supports concurrent read-write mounts on shared iSCSI SAN storage—validated with 2, 4, 8, and 16 nodes—or on dual-primary DRBD, with network-based or SCSI COMPARE AND WRITE locking. MeshXFS provides fencing and survivor-led journal recovery, is packaged as a DKMS module for Proxmox VE 9, RHEL 9, Ubuntu 24.04, Ubuntu 26.04, Debian 12 and Debian 13, and includes a Proxmox VE storage plugin.
On a system, MeshXFS goes by its short name mxfs: the mxfs.ko module, mount -t mxfs, the tools mkfs.mxfs, chk_mxfs (fsck.mxfs), resize_mxfs and mxfs_admin, and the mxfs and pve-storage-mxfs packages.
A shared-LUN clustered filesystem written entirely by AI.
Every line of the clustering, coordination, distributed-lock, fencing, and on-disk-envelope code in this repository was written by AI — not by a human using an AI tool. Claude Opus, running as an autonomous coding agent in Anthropic's Claude Code, wrote a large share of it, consulting Claude Fable, GPT, Gemini and Astra on the hardest problems. MeshXFS starts from the upstream Linux kernel XFS code, written by the XFS developers, and the AI has since modified much of it heavily (see "How this was built"). Everything that turns XFS into a filesystem many machines can mount at once is AI-authored.
What changed for someone upgrading from 0.90.129:
Released configurations and what 0.90.130 was validated on:
The test rig is Ubuntu 24.04 (kernel 6.8.0-101-generic). "Installed packages" means the release packages on two nodes of each supported platform: Proxmox VE 9 (kernels 6.17 and 7.0), RHEL 9.8, Ubuntu 24.04, Ubuntu 26.04, Debian 12 and Debian 13. Every host in a cluster needs the same MeshXFS version; hosts on different kernels are fine.
What changed for someone upgrading from 0.90.124, the last published release (0.90.125 to 0.90.128 were never published; their work ships here):
Released configurations and what 0.90.129 was validated on:
The test rig is Ubuntu 24.04 (kernel 6.8.0-101-generic). "Installed packages" means the release packages on two nodes of each released platform: Proxmox VE 9 (kernels 6.17 and 7.0), RHEL 9.8, Ubuntu 24.04 and Debian 13. Every host in a cluster needs the same MeshXFS version; hosts on different kernels are fine.
What changed for someone upgrading from 0.90.116, the last published release:
Still open: under heavy load on slow disks, a VM write that needs new space can still wait about a second for a slow answer from the other host of a DRBD pair (D-REMOTE-NOQUEUE-AG-TRYLOCK-WAITS-A-FULL-ATTEMPT-PER-AG-UNDER-ILOCK). No write fails and no data is at risk.
Released configurations and what 0.90.124 was validated on:
The test rig is Ubuntu 24.04 (kernel 6.8.0-101-generic): one iSCSI path per node for direct, two for mpath (multipath-tools 0.9.4, the settings of tools/mpath_settings.sh). The path-fault rows (docs/mpath-verification.md) run under load on every node: one path lost and returned and then the other, a whole storage network lost on every node at once, a flapping path, a mount and an unmount on one path, a node killed while the survivors are on one path, a fenced node whose dead path comes back, every path lost, a node that withdraws, and a lock command the target applied whose answer never arrived. "Installed packages" means the release packages on two nodes of each of Proxmox VE 9, RHEL 9.8, Ubuntu 24.04 and Debian 13; those rounds attach on one path, so mpath is claimed on Ubuntu 24.04 only. Not covered for mpath: a second target or storage controller, Fibre Channel or SAS, more than two paths, other multipath settings, and automatic failback to a preferred path group. Every host in a cluster needs the same version.
For 2/net/mesh/drbd, a VM's disk writes no longer freeze on slow disks. With seven AlmaLinux installs at once on two physical Proxmox VE 9 hosts whose DRBD sat on 2011-era SATA SSDs, a guest write that needed new space could hold its disk image locked for 30 s to a minute. Its lock request had asked only to try, but it waited on the cluster's lock-ownership records, which were being written to the slow disk. Every other write to that image waited behind it, and the host kernel reported hung tasks. No write failed and no data was lost. 0.90.116's try never touches the disk and never waits: when its answer is not ready, it says so at once, the write takes space elsewhere, and a background worker readies the answer for next time. On the same disks and load, none of the slow waits remained, no task hung, and all seven installs finished (CHANGELOG.md, 0.90.115 and 0.90.116).
A host whose MeshXFS mount is shut down and then rejoins no longer hangs or stalls. When a host's heartbeat writes stop long enough, the other host declares its mount dead, recovers it, and the host's guard unmounts and remounts it with no restart. On two physical Proxmox VE 9 hosts that rejoin had two faults when the rejoining host held the lower heartbeat slot:
No data was lost in either. Both were found by forcing the dropout on purpose (tests/pve_pair_failover.sh withdraw-p0), as happened for real on 2026-10-06 when a swapping host's heartbeat writes stalled. Everyday use did not reach either one. CHANGELOG.md has what was measured, 0.90.108 to 0.90.114.
8/net/mesh/direct, 16/net/mesh/direct and every mpath configuration were not tested on this release and are not claimed.
2/net/mesh/drbd is released again: two hosts, each with its own local disk, replicated by DRBD dual-primary, with MeshXFS mounted read/write on both and no shared storage at all. It was withdrawn on 2026-10-06, after eight concurrent VM installs on two physical Proxmox VE 9 hosts shut both hosts' MeshXFS mounts down and gave the guests I/O errors. What that load found was fixed (CHANGELOG.md, 0.90.55 to 0.90.107), and this release was verified on those same two hosts as well as on the test rig:
What it is not: fast under heavy writes on modest disks. Both disks write everything and every write waits for both, so a pair has one disk's write capacity. On the physical pair one AlmaLinux install on MeshXFS takes 1.4–1.9 times as long as on a local disk, and XFS on the same DRBD takes about three quarters of MeshXFS's time. Seven installs at once did not finish in 43 minutes, and under that load one heartbeat write took 8.6 s (the lease is 60 s). A lone sequential writer gets about three quarters of what XFS gets on the same DRBD: MeshXFS bounds its own writes in flight on DRBD so that its coordination writes are never starved. Size the VMs on a pair to its disks. Each of these is an open record in the defect queue.
Also fixed, for every configuration: unmounting a filesystem that had been shut down could free the lock manager under a background log task and crash the node. The six direct configurations that no open defect blocks, and all four platforms, were verified again on this build. The eight mpath configurations and 8/net/mesh/direct and 16/net/mesh/direct are not claimed by this release, because defects found after 0.90.51 reach them; see "Released in an earlier version" below.
The mpath attachment is released at 2, 4, 8 and 16 nodes with both lock managers: {2,4,8,16}/net/mesh/mpath and {2,4,8,16}/disk/caw/mpath. Every node reaches the shared LUN through a dm-multipath map over two iSCSI paths, each path on its own NIC and its own storage network. Each of the eight configurations passed the 31-row suite the direct configurations pass, and then the path-fault rows of docs/mpath-verification.md, all under load on every node: one path lost and returned and then the other, a whole storage network lost on every node at once, a flapping path, a mount and an unmount on one path, a node killed while the survivors are on one path, a fenced node whose dead path comes back, every path lost, a node that withdraws, and on disk/caw a lock command the target applied whose answer never arrived. A row passes only with no failed operation on any node, no node lost that the row did not kill, every stall under the bound, and every acknowledged file read back by checksum from another node.
Making those rows pass took changes to the filesystem, not only to the rig: a node now registers its fencing key on each path itself and follows its paths as they come and go, the lone survivor's fence works with one registration per path, a disk/caw lock swap whose answer was lost is recognised as landed, and several faults that are not specific to multipath were found and fixed on the way (a lock wait that lost its registration, a refused lock upgrade that starved behind its own node's readers, a healthy node shutting down behind a stalled peer, a survivor opening a dead node's files as directories, a lookup failing when its name was removed under it). CHANGELOG.md has each with what was measured.
What this covers: iSCSI; two paths on separate networks to ONE target; the multipath and initiator settings in tools/mpath_settings.sh, which are the ones to use; the test rig's platform, Ubuntu 24.04 with kernel 6.8.0-101-generic and multipath-tools 0.9.4. What it does not: a second target or storage controller, Fibre Channel or SAS, more than two paths, other multipath settings, automatic failback to a preferred path group, and the other three platforms, whose package rounds attach on one path.
The eight direct configurations and all four platforms were verified again on this build.
16/net/mesh/direct and 16/disk/caw/direct are released. Sixteen nodes mounting one LUN, with either lock manager, passed the same 31-row suite the smaller clusters pass — data integrity, coherency, crash and fence recovery, pace against native XFS — on the build this release ships, and the 2-, 4- and 8-node configurations and all four platforms were verified again on it.
Two defect records stood between 8 nodes and 16. One was disproved by measurement. The other was a real fault in the disk/caw lock manager's clean-up after an abandoned lock wait, found by reading the code and never seen on the rig: the clean-up could act on a lock slot that had since been reused for a different lock. The check that stops it from releasing that other lock was already in the tree; this release adds a deterministic test for it and fixes what the test then showed — when the slot had been reused, the abandoned wait's own registration was left behind, which stalls every other node's readers of that lock. See CHANGELOG.md.
Two nodes, a local disk each, no SAN, no iSCSI target, no third server. DRBD replicates the two disks synchronously (protocol C) with both nodes Primary at once, and MeshXFS mounts /dev/drbd0 read/write on both. The configuration is 2/net/mesh/drbd. This is the smallest possible MeshXFS cluster: two Proxmox hosts with local NVMe can share one filesystem for VM images and live migration without buying storage.
DRBD has no SCSI underneath, so it has neither persistent reservations (how MeshXFS fences a dead node) nor COMPARE AND WRITE (how it claims heartbeat and recovery records). MeshXFS supplies both from the attachment itself:
Measured on the rig (two nodes, DRBD 8.4.11, Ubuntu 24.04, the 0.90.41 module built from the tree): the cluster suite's rows all passed, the fault rows and crash_audit included; a power cut of both nodes recovered with every file of both nodes intact, also when the recovery's completion was refused partway and finished by a resume; a recovery whose owner died was taken over and finished, by the owner's next boot and by the other node, every file intact; a node crash with a writer running was fenced, certified and recovered in 84 s with every fsynced file of both nodes intact; a link cut settled in 11 s and a split had exactly one winner in 11 s; unmount/remount cycles, a crash-cut re
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。