Modern Filesystems: Journaling vs Copy-on-Write, the Kernel Write Path, and Data Correctness

When engineers discuss storage and file systems, the conversation often starts and stops with the classics: ext4 on Linux, NTFS on Windows, and perhaps XFS for large database servers. For decades, the dominant answer to filesystem reliability was straightforward: journaling.

However, the hardware and operational landscape of computing has changed drastically over the last fifteen years. We transitioned from spinning magnetic platters to high-throughput NVMe NAND flash, storage-class persistent memory, and petabyte-scale distributed object storage. At the same time, requirements evolved from simple crash resilience to instantaneous snapshots, zero-overhead cloning, native software pooling, and cryptographic protection against silent bit rot.

This shift gave rise to a fundamental architectural showdown: Journaling vs. Copy-on-Write (CoW), as well as specialized paradigms like Flash-Friendly Log-Structured File Systems (LFS) and Distributed Cloud-Native POSIX layers.

To understand why modern systems behave the way they do, we need to trace the exact journey data takes: from the moment an application invokes a system call, down through kernel page caches and block drivers, all the way to physical flash cells. We will examine how both journaling and CoW architectures prove correctness, eliminate torn writes, and protect running servers from catastrophic crashes caused by bad reads and misdirected writes.

The Anatomy of a Write: From Application to Silicon

#

Before comparing journaling and Copy-on-Write, we must understand the baseline I/O path common to all standard POSIX filesystems on Linux.

+-------------------------------------------------------------------------+
|                              Application                                |
|   Userspace Buffer (e.g., char buf[4096])                               |
+------------------------------------+------------------------------------+
                                     |
                                     | System Call: write(), pwritev2()
                                     v
+------------------------------------+------------------------------------+
|                         Virtual Filesystem (VFS)                        |
|   Resolves file descriptors, permissions, directory navigation (dentry) |
+------------------------------------+------------------------------------+
                                     |
                                     v
+------------------------------------+------------------------------------+
|                         Linux Page Cache                                |
|   Pages marked "dirty" in RAM. Write returns immediately to app.       |
+------------------------------------+------------------------------------+
                                     |
                                     | Asynchronous flush (kworker) OR
                                     | Synchronous invocation (fsync/fdatasync)
                                     v
+------------------------------------+------------------------------------+
|                      Concrete Filesystem Layer                          |
|   Journaling (ext4 / XFS)     OR     Copy-on-Write (ZFS / Btrfs)        |
+------------------------------------+------------------------------------+
                                     |
                                     v
+------------------------------------+------------------------------------+
|                         Generic Block Layer                             |
|   Constructs bio structures, manages I/O scheduler (mq-deadline/none)   |
+------------------------------------+------------------------------------+
                                     |
                                     v
+------------------------------------+------------------------------------+
|                       Device Driver (NVMe / SCSI)                       |
|   Submits command queues to hardware controller                         |
+------------------------------------+------------------------------------+
                                     |
                                     v
+------------------------------------+------------------------------------+
|                     Physical Storage Hardware                           |
|   Volatile On-Disk DRAM Cache  ==[NVMe FLUSH / FUA]==>  NAND Flash Cells |
+-------------------------------------------------------------------------+

The System Call Boundary

#

When an application wants to persist bytes, it typically begins with one of several system calls:

1. write(int fd, const void *buf, size_t count) / pwrite(...):
   The application passes a pointer to user memory. The kernel switches from user mode to kernel mode via a software trap (syscall), navigates the Virtual Filesystem (VFS) to locate the target inode, allocates or locates memory pages in the Linux Page Cache, and copies the user buffer into those pages using copy_from_user().
2. The “Dirty” Page:
   Crucially, standard write() calls do not touch the physical disk. The page in RAM is simply marked as “dirty”. The system call returns 0 (success) almost instantly.
3. Flusher Threads:
   In the background, kernel threads (kworker/flush) wake up periodically (governed by /proc/sys/vm/dirty_writeback_centisecs and /proc/sys/vm/dirty_ratio) to write dirty pages to disk asynchronously.

The Problem: Power Loss and Memory Volatility

#

If the machine loses power while data sits only in the Page Cache, that data is permanently lost. Furthermore, even after the kernel issues block requests to the disk controller, modern SSDs and HDDs feature their own internal volatile DRAM write caches. The drive may report that a write has finished when data has only reached its internal volatile RAM.

To guarantee that data actually survives a sudden blackout, applications must explicitly request durability using synchronization system calls:

• fsync(int fd): Instructs the filesystem to flush all dirty data pages and all modified metadata (file size, modification timestamps, extents) associated with the file to non-volatile storage, issuing hardware-level flush commands (SYNCHRONIZE CACHE on SCSI/SATA or NVMe Flush).
• fdatasync(int fd): Like fsync(), but skips flushing non-essential metadata changes (such as access or modification timestamps) unless they are required to locate the data blocks (such as a change in file size). Databases like PostgreSQL and MySQL heavily rely on fdatasync() to eliminate redundant metadata writes during transaction commits.
• open(..., O_SYNC | O_DSYNC): Forces every subsequent write() to block until the data and required metadata are completely settled on persistent physical media.
• open(..., O_DIRECT): Bypasses the Linux Page Cache entirely. User buffers must be memory-aligned (typically 4 KB or 512-byte boundaries). The block layer reads/writes directly from userspace buffers to the storage controller. However, O_DIRECT does not guarantee the drive’s volatile write cache is flushed; an explicit fsync() or O_SYNC is still required.

Now, how do different filesystem paradigms handle the moment fsync() is called? That is where Journaling and Copy-on-Write diverge completely.

The Deep Dive: How Journaling Handles the Write Path

#

Journaling file systems—exemplified by Linux’s ext4 (via its Journaling Block Device layer, JBD2) and Silicon Graphics’ XFS—maintain an in-place overwrite model. File blocks are tied to fixed logical block numbers (LBAs).

The In-Place Overwrite Problem

#

In early un-journaled filesystems like ext2, updating an existing file block required writing directly to its allocated disk location. If the system crashed halfway through writing an 8 KB block (or when updating the directory table and inode), the on-disk structures became corrupt. The operating system could not tell what state was left on disk without running fsck, which scanned every inode, directory, and allocation bitmap on the entire partition.

The Journaling Data Path and JBD2

#

Journaling solves this by introducing a dedicated, contiguous circular log on disk (the journal).

When an application writes to a file on ext4 with the default data=ordered mode and calls fsync():

ext4 (data=ordered) fsync() Workflow:

1. Write File Data to Permanent Blocks (In-place)
   [ Dirty Page in RAM ] ======================> [ Permanent Data Blocks on Disk ]
                                                              ||
2. Hardware Barrier / Flush                                   \/ (Wait for completion)
   [ NVMe Flush Command ] <===================================++

3. Append Metadata Changes to Journal (JBD2)
   +---------------------------+----------------------------+
   | Journal Descriptor Block  | Inode / Extent Tree Update |
   +---------------------------+----------------------------+

4. Write Commit Record to Journal with Checksum
   +--------------------------------------------------------+
   | Journal Commit Block (CRC32 Checksum of Transaction)   |
   +--------------------------------------------------------+
                                                              ||
5. Hardware Barrier / Flush                                   \/ (Wait for completion)
   [ NVMe Flush Command ] <===================================++
   
6. fsync() returns success to application!

--- (Later, asynchronously during "Checkpointing") ---
7. Metadata written to permanent inode table / allocation bitmaps in-place.
8. Transaction removed from journal ring buffer.

1. Flush Data First: In data=ordered, the filesystem issues block I/O requests for the dirty data blocks directly to their permanent locations on disk.
2. Issue Flush Barrier: The kernel waits until the hardware controller confirms that the raw data blocks are settled on non-volatile media.
3. Write Journal Descriptor: The metadata changes (e.g., the modified inode, extent tree changes, block bitmaps) are packaged into a running JBD2 transaction and written sequentially to the journal.
4. Commit Block with Checksum: JBD2 writes a final Commit Block containing a CRC32 checksum calculated over the entire journal transaction.
5. Final Storage Barrier: Another hardware flush barrier is issued to ensure the commit block has hit physical media.
6. Return to App: Once the commit block is persistent, fsync() unblocks and returns 0.
7. Checkpointing: At a later time (often during background memory reclaim), the kernel writes those metadata modifications to their permanent in-place locations (the actual inode tables and allocation bitmaps) and reclaims space in the circular journal log.

Proving Correctness & Crash Recovery in Journaling

#

How does journaling prevent corruption if the power cord is yanked?

• Case 1: Power loss during Data Write (Step 1): The journal contains no commit record for this transaction. Upon reboot, the filesystem sees no committed transaction. The metadata still points to the old, un-updated version of the file. The half-written data blocks are abandoned. No corrupt state is exposed.
• Case 2: Power loss during Journal Write (Step 3): The journal has an incomplete transaction without a valid commit block. On mount, JBD2 scans the journal, detects an uncommitted transaction, and safely discards it.
• Case 3: Power loss after Commit (Step 5), before Checkpoint (Step 7): The commit block is safely on disk. On mount, JBD2 performs Journal Replay: it reads the transaction from the journal and immediately reapplies the metadata changes to the permanent inode tables. The filesystem recovers to a fully consistent state in milliseconds, without running fsck.

The Vulnerability: Torn Writes on User Data

#

While journaling protects filesystem metadata integrity, standard journaling filesystems do not protect user data blocks against torn writes.

Consider a database like PostgreSQL or MySQL writing an 8 KB or 16 KB page to a disk whose physical sector or atomic write boundary is 4 KB:

• If power fails halfway through writing an 8 KB page, the drive writes the first 4 KB sector, but the second 4 KB sector remains old data!
• The filesystem does not notice: the metadata is consistent, the inode is intact, and the filesystem mounts without error.
• But when the database engine starts up and reads that page, it encounters a scrambled half-new, half-old page—a torn write.

Because ext4 and XFS overwrite data in-place and only journal metadata by default, they cannot detect or fix torn writes in user data.

This single limitation is why major databases were forced to invent complex application-level workarounds:

• MySQL InnoDB: Implemented the Doublewrite Buffer (writing pages twice: first to a sequential doublewrite buffer, then to the permanent tablespace).
• PostgreSQL: Implemented Full Page Writes (full_page_writes = on), writing the complete 8 KB image of every modified page to the Write-Ahead Log (WAL) after every checkpoint.

The Deep Dive: How Copy-on-Write Handles the Write Path

#

Copy-on-Write (CoW) filesystems—such as ZFS and Btrfs—completely eliminate in-place overwriting. Their fundamental law is: Never overwrite live, referenced blocks.

Instead of updating an existing block, the filesystem always writes modifications into brand-new, freshly allocated space. It then updates parent pointers in a tree hierarchy (a Merkle Tree) until reaching a single root anchor.

The CoW Data Path: Merkle Trees and Atomic Root Switches

#

In a CoW filesystem, all files, directories, extent records, and metadata are represented as nodes in a tree.

Copy-on-Write (Merkle Tree) Write Architecture:

                 [ Root Anchor / Uberblock ]
                              |
                              v
                  [ Parent Directory Node ]
                     /                \
          [ Subdir Node ]        [ Inode Node ]
                                  /          \
                          [ Block A ]    [ Block B ]

When an application calls write() and modifies Block A, followed by fsync():

CoW fsync() Step-by-Step Execution:

1. Allocate Fresh Blocks & Calculate Checksum
   - Allocate new location on disk for [ Block A' ]
   - Calculate cryptographic/Fletcher checksum of [ Block A' ]
   - Write [ Block A' ] to physical storage

2. Propagate Upwards: Allocate New Parent Node
   - Allocate new location for [ Inode Node' ]
   - Point to original [ Block B ] (unchanged!)
   - Point to [ Block A' ] and store its Checksum inside [ Inode Node' ]
   - Calculate checksum of [ Inode Node' ]
   - Write [ Inode Node' ] to physical storage

3. Propagate to Root:
   - Allocate new location for [ Parent Directory Node' ]
   - Point to [ Inode Node' ] with its checksum
   - Write [ Parent Directory Node' ] to physical storage

4. Atomic Root Flip:
   - Write new [ Uberblock / Superblock ] pointing to [ Parent Directory Node' ]
   - Hardware Barrier / Flush

5. fsync() returns success!

Visualizing the Result of the Atomic Root Flip:

Old Generation (Untouched!):
[Root v1] ---> [Dir Node] ---> [Inode Node] ---> [Block A] (Old)
                                            ---> [Block B] (Shared)

New Generation (Active):
[Root v2] ---> [Dir Node'] ---> [Inode Node'] -> [Block A'] (New)
                                            ---> [Block B] (Shared)

Proving Correctness: Immunity to Torn Writes

#

Because of this recursive allocate-on-write process, CoW filesystems achieve mathematical correctness without an intent journal for data integrity:

1. Complete Immunity to Torn Writes:
   If power fails while the drive is writing Block A', Inode Node', or Parent Directory Node', the write simply aborts. The physical drive might contain a torn, half-written block at the newly allocated address. It does not matter. The root anchor (Uberblock v1) still points to the old, uncorrupted generation tree (Block A). The filesystem mounts Root v1. The half-written blocks in unallocated space are simply identified as unreferenced free space and reclaimed. The live filesystem state is 100% consistent.
2. Atomic Root Anchors:
   The only in-place write that ever occurs in a CoW filesystem is the root anchor. ZFS accomplishes this using an array of 128 separate Uberblocks arranged as a circular ring. Each uberblock contains a monotonic transaction number (TXG) and its own internal checksum. When committing, ZFS writes to the next uberblock slot in the ring. On mount, ZFS scans all 128 uberblocks and selects the one with the highest transaction number whose internal checksum is valid. If a crash tore an uberblock write, that slot is rejected, and ZFS mounts the immediately preceding valid transaction.

Preventing Silent Data Corruption (Bit Rot) & Bad Reads

#

Traditional filesystems trust the underlying hard drive. If a cosmic ray flips a bit on the platter, or if a failing SSD controller firmware writes data to the wrong sector (a phantom write or misdirected write), ext4 will happily read that corrupt data and hand it to PostgreSQL or your application.

The application may crash, or worse, silently ingest corrupted numbers into financial ledgers.

CoW filesystems protect against this via Merkle Tree Checksum Verification:

Parent-Stored Checksum Architecture:

[ Parent Inode Pointer ]
  - Pointer Address: Block 0xDEADBEEF
  - Stored Checksum: 0x8F3C1A... (Calculated over Block A' at write time)
          |
          v
[ Block 0xDEADBEEF (Block A') ]
  - Data payload

1. Checksum verification on every read: When the OS reads Block A', the filesystem calculates the checksum of the bytes coming off the disk wire and compares it against the checksum stored in the parent node.
2. Detecting Misdirected Writes: If the drive controller accidentally wrote Block A' to block 0xCAFEBABE instead of 0xDEADBEEF, reading 0xDEADBEEF will return whatever old garbage was there. The checksum will fail instantly.
3. Self-Healing on Mirror / Parity (RAID-Z / Btrfs RAID1):
   If the checksum verification fails, the filesystem does not panic or crash the server. Instead:
   • It intercepts the read error in the filesystem driver.
   • It reads the alternate copy from the mirror drive or calculates the missing block from RAID-Z parity.
   • It verifies that the alternate copy’s checksum matches the parent pointer.
   • It serves the clean data to the application (the application never sees an error).
   • In the background, it rewrites the repaired block back to the corrupted drive, healing the disk on the fly!

Self-Healing Read Path in CoW:

App calls read()
       |
       v
Read Primary Block A' ----> Checksum Mismatch! (Bit Rot / Torn Read)
                                   |
                                   v (Do NOT return bad data!)
                            Read from Mirror Drive / Parity
                                   |
                                   v Checksum OK!
                        +----------+----------+
                        |                     |
                        v                     v
                 Return Clean Data    Rewrite Clean Data
                 to Application!      to Primary Drive (Auto-Heal)

Detailed Trade-Offs: Journaling vs. Copy-on-Write

#

With all these protections, why doesn’t every server run Copy-on-Write?

The very mechanics that give CoW its superpowers introduce severe architectural trade-offs.

+------------------------+------------------------------------+------------------------------------+
| Architectural Aspect   | Journaling (ext4, XFS)             | Copy-on-Write (ZFS, Btrfs)         |
+------------------------+------------------------------------+------------------------------------+
| Write Strategy         | In-place overwrite                | Allocate-on-write (out-of-place)   |
| Data Integrity         | Metadata only (by default)         | End-to-end Merkle checksums        |
| Torn Write Handling    | Relies on DB doublewrite buffers   | Immune (old tree preserved)        |
| Bit Rot Protection     | None                               | Automatic detection & self-healing |
| Snapshots              | Slow / external (LVM)              | Instantaneous & zero space penalty |
| Random Overwrite Perf  | Predictable, constant sequentiality| Suffers severe fragmentation       |
| Write Amplification    | Low (metadata-only journal)        | High (Merkle tree pointer bubbling)|
| Memory Footprint       | Minimal (standard page cache)      | High (B-trees, ARC, deduplication) |
| Running Out of Disk    | Standard behavior (ENOSPC)         | Risk of deadlocks / write failure  |
+------------------------+------------------------------------+------------------------------------+

1. The Random Overwrite Penalty & Database Performance

#

Databases like PostgreSQL, MySQL, and hypervisors managing VM disk images (.qcow2, .vmdk) allocate large files (tens or hundreds of gigabytes) and continuously overwrite small 4 KB or 8 KB pages inside them.

• On ext4/XFS: The block number remains static. The write occurs in-place. The file remains 100% contiguous on physical disk. Sequential scans continue to stream at maximum hardware bandwidth.
• On Btrfs/ZFS: Every modified 4 KB page is written to a new, arbitrary physical location. Over days and weeks, a single database file becomes fragmented into hundreds of thousands of discontinuous extents scattered across the disk.
• The Result: Read latency spikes drastically. To combat this, administrators running databases on CoW filesystems must either disable CoW for the database directory (e.g., chattr +C on Btrfs, which disables checksumming and snapshots!), or configure high recordsize alignments and run frequent defragmentation jobs.

2. Write Amplification and Tree Cascading

#

In a journaling filesystem, modifying an existing block requires writing that single block, plus appending a tiny metadata journal entry.

In a CoW filesystem, modifying a single leaf block requires:

1. Writing the leaf data block.
2. Writing the new parent pointer block.
3. Writing the grandparent pointer block.
4. Writing all intermediate nodes up to the tree root.

While CoW filesystems batch these modifications in RAM and flush them in transaction groups (TXGs) every few seconds, synchronous workloads that call fsync() frequently force tree flushes, causing significant write amplification that can wear out consumer SSDs prematurely.

3. Synchronous Writes: Intent Logs in CoW (ZIL and Tree-Log)

#

Because bubbling a complete Merkle tree to the root anchor on every single microsecond fsync() would destroy write performance, modern CoW filesystems had to bring back… a specialized journal!

• ZFS ZIL (ZFS Intent Log): When an application calls fsync(), ZFS does not commit an entire transaction group to the main Merkle tree. Instead, it writes a lightweight transaction record to an intent log called the ZIL. If you add a dedicated, high-speed power-loss-protected NVMe drive for this, it is called a SLOG (Separate Intent Log). On normal operation, this log is never read; it is only replayed if the system crashes before the next main transaction group flushes.
• Btrfs Log-Tree (tree-log): Btrfs maintains sub-trees dedicated to recording uncommitted fsync modifications for specific files, avoiding a full filesystem-wide transaction commit on every sync.

Flash-Native & Log-Structured File Systems (LFS)

#

Both traditional journaling and CoW filesystems were originally conceived under the abstraction of spinning magnetic platters. However, raw NAND Flash memory (SSDs, eMMC, UFS) behaves very differently:

1. Pages vs. Blocks: Data can be read and written in small pages (e.g., 4 KB to 16 KB), but cannot be overwritten until a much larger erase block (typically 2 MB to 8 MB) is completely erased.
2. Wear Leveling: Each flash cell can only endure a finite number of Program/Erase (P/E) cycles before failing.
3. The FTL: SSDs contain a controller running a Flash Translation Layer (FTL) that handles garbage collection and remapping in secret.

When filesystems perform scattered in-place overwrites or haphazard CoW allocations, they force the SSD controller to perform heavy internal garbage collection, causing high write amplification and degrading drive lifespan.

F2FS (Flash-Friendly File System)

#

To address this, Samsung developed and contributed F2FS to the Linux kernel.

F2FS is an implementation of a Log-Structured File System (LFS) specifically tuned for flash media:

Log-Structured Append Model (F2FS):
[ Append Write 1 ] -> [ Append Write 2 ] -> [ Append Write 3 ] ---> (Sequential Stream)

Instead of scattering updates, F2FS treats storage segments as sequential append logs. All modifications—data and metadata alike—are streamed sequentially into active allocation segments.

Key architectural traits of F2FS:

• Segment Cleaning: Dedicated background cleaning threads coalesce fragmented valid blocks from older segments into continuous free blocks, matching the erase-block geometry of flash drives.
• Node & Data Separation: Keeps metadata nodes separate from data logs to minimize garbage collection overhead.
• Roll-Forward Recovery: Leverages an efficient checkpointing mechanism paired with direct node addressing for instant crash consistency.

Today, F2FS is the standard default filesystem on billions of Android smartphones and handheld gaming devices, where NAND flash write endurance, app launch latency, and random I/O responsiveness are paramount.

Cloud-Native & Distributed Filesystems

#

As workloads transitioned to Kubernetes clusters and multi-region clouds, local block devices became insufficient for applications needing shared persistent storage.

Traditional network filesystems like NFS and CIFS/SMB suffer from single-point-of-failure bottlenecks, locking conflicts, and poor scalability over high-latency links.

Decoupled Metadata: The JuiceFS Model

#

Modern cloud-native filesystems decouple the filesystem into two distinct tiers:

1. Metadata Engine: Uses an ultra-fast in-memory or distributed transactional database (e.g., Redis, TiKV, or SQLite) to manage directory trees, permissions, inodes, and file locks.
2. Data Storage: Chunks file data payloads into binary blocks and stores them directly in cost-effective cloud object storage (Amazon S3, Google Cloud Storage, or MinIO).

JuiceFS Architecture:

             POSIX Client (FUSE / CSI Driver)
                 /                      \
      (Metadata Operations)        (Data Chunks)
               /                          \
              v                            v
   +---------------------+      +---------------------+
   |   Metadata Engine   |      |    Object Storage   |
   | (Redis / TiKV / SQL)|      |  (S3 / GCS / MinIO) |
   +---------------------+      +---------------------+

This model provides full POSIX compliance to containers and HPC workloads without needing to manage physical hard disks or RAID arrays, scaling transparently to petabytes.

Cluster Filesystems: CephFS and HPC Storage

#

For on-premises data centers and high-performance computing (HPC):

• Ceph (CephFS): Distributes file metadata across dedicated Metadata Servers (MDS) while striping data blocks across a peer-to-peer storage cluster using the CRUSH algorithmic placement engine.
• Lustre / BeeGFS: Parallel filesystems engineered for supercomputing clusters, decoupling Metadata Targets (MDT) from Object Storage Targets (OST) to deliver terabytes-per-second of aggregate throughput for machine learning and scientific simulations.

Storage-Class Memory & Direct Access (DAX)

#

At the extreme performance frontier lies Storage-Class Memory (SCM) and CXL-attached Persistent Memory. These devices offer nanosecond-level access latencies and are byte-addressable directly over the CPU memory bus, rather than block-addressable over PCIe/SATA.

Traditional filesystems route every read and write through the Linux kernel’s Page Cache and block I/O scheduling layers. For persistent memory, this kernel overhead is far slower than the physical medium itself.

• Direct Access (DAX): A mount capability in filesystems like ext4 and XFS (-o dax) that bypasses the OS page cache entirely. Memory mapping (mmap) maps the storage media directly into the process’s virtual address space, allowing applications to read and write with CPU MOV instructions.
• NOVA (Non-Volatile Memory Accelerated): A specialized log-structured filesystem designed explicitly for byte-addressable NVRAM, maintaining separate per-core inode logs to eliminate lock contention on massively multicore servers.

Summary Decision Guide: What Should You Use?

#

1. For Production Relational Databases (PostgreSQL, MySQL, Oracle) & Virtualization Hosts (KVM/Proxmox with raw images):
   Choose ext4 or XFS. The predictable, in-place write path avoids random-write fragmentation spirals. Let the database’s internal WAL and doublewrite mechanisms manage page-level correctness.
2. For Backup Repositories, File Servers, NAS, and Homelabs:
   Choose ZFS or Btrfs. The end-to-end Merkle tree checksums, automatic self-healing, instant atomic snapshots, and transparent compression provide unmatched data durability.
3. For Smartphones, Tablets, and Embedded NAND Flash:
   Choose F2FS. Its log-structured segment cleaner works in harmony with flash memory erase blocks to extend device life and sustain snappy interactive performance.
4. For Kubernetes Clusters, Shared ML Training Datasets, and Cloud Pipelines:
   Choose JuiceFS or CephFS. Decoupling low-latency distributed metadata from commodity object storage allows you to mount petabyte-scale POSIX filesystems without managing complex hardware disk arrays.