# How file systems work Source: https://archil.com/guides/how-file-systems-work How files are named, written, cached, shared, copied, and billed, from the kernel to the cloud, taught one step at a time with figures you can drive. ## Contents 1. [From a path to an inode](https://archil.com/guides/how-file-systems-work/from-a-path-to-an-inode.md): How a POSIX file system names, finds, and changes a file, from inodes and directories through path lookup, descriptors, rename, links, and permission bits. Then how an object store’s keys map onto that model, and the four places where they can’t. 2. [Where fsync returns from](https://archil.com/guides/how-file-systems-work/where-fsync-returns-from.md): How one write travels from a program’s memory to a drive’s flash and then to other machines, what `write()`, `fsync()`, `fdatasync()`, and `O_DIRECT` each wait for, and why replicated storage acknowledges where it does. By the end, you can name the layer any storage system acknowledges from, and the failures that choice survives. 3. [Anatomy of a file create over a network](https://archil.com/guides/how-file-systems-work/anatomy-of-a-file-create.md): What a program asks for when it creates one small file, why a network file system has to ask a server before it can answer, and how callbacks, leases, and delegations let a client answer from its own memory once it holds the right to. 4. [One file, many caches](https://archil.com/guides/how-file-systems-work/one-file-many-caches.md): What a second machine sees after you write a file, and who gets to write it next. By the end you can tell linearizable, close-to-open, and eventual consistency apart, explain how a cache learns it’s stale, and say why leases exist and why locks across machines are harder than they look. 5. [A file system on an object store](https://archil.com/guides/how-file-systems-work/a-file-system-on-an-object-store.md): How a tree of files, with its modes, links, and renames, gets laid onto a flat map of immutable objects. By the end, you can read any product’s bucket layout and say what it gives up, what it costs to rename and list, and what happens when someone else writes to the bucket. 6. [Copy-on-write, block by block](https://archil.com/guides/how-file-systems-work/copy-on-write.md): How a file system can copy a terabyte by writing one block, why the same design survives a power cut, and why deleting a snapshot takes more work than taking one. By the end you’ll be able to explain snapshots, clones, and their space accounting from the blocks up. 7. [How storage is metered](https://archil.com/guides/how-file-systems-work/how-storage-is-metered.md): Why a provider sells storage as reserved capacity or as bytes stored, what a GB-month measures, why cold data costs less to keep and more to read, and why, for agents, an idle machine costs more than the bytes it holds. ## 1. From a path to an inode Source: https://archil.com/guides/how-file-systems-work/from-a-path-to-an-inode · Part of How file systems work · Facts checked 2026-11-04 **In this chapter:** How a POSIX file system names, finds, and changes a file, from inodes and directories through path lookup, descriptors, rename, links, and permission bits. Then how an object store’s keys map onto that model, and the four places where they can’t. If most of your storage has been S3, you probably picture a file as a key and some bytes. You `PUT` the whole thing, you `GET` it back, and the key is its full path. A POSIX file system works differently. It finds a file one name at a time, changes it a few bytes at a time, and renames it without touching its contents. Those differences decide which programs still work when a file system is built on top of a bucket. In 1974, Dennis Ritchie and Ken Thompson described the Unix file system in a paper that still reads as a description of Linux. One sentence in it is the subject of this page: “The purpose of an open or create system call is to turn the path name given by the user into an i-number by searching the explicitly or implicitly named directories.” (Ritchie and Thompson, [“The UNIX Time-Sharing System”](https://dsf.berkeley.edu/cs262/unix.pdf), _Communications of the ACM_ 17(7), July 1974, section 4. An i-number is what this page calls an inode number.) The following sections build that sentence up from a bare disk, one idea at a time, and then hold an object store up against it. **Two pictures of a file system** On the left, a whole-file API, such as S3’s `PutObject` and `GetObject` or a `writeFile` and `readFile` pair, makes one call per whole file and finds the file by its path. On the right, what the kernel sends: it looks up `src` in the repository’s directory, inode 31, and gets inode 12, looks up `app.ts` in inode 12 and gets inode 87, opens it, and writes 512 bytes at offset 4096. A rename changes a name in directory 12, and none of the file’s bytes move. ### Blocks and inodes A disk offers a small interface: read block _n_, and write block _n_. The blocks have a fixed size, often 4 KiB, and the disk knows nothing about files, names, or lengths. A file system’s first job is bookkeeping: which blocks belong to which file, how long the file is, and who may read it. Unix keeps that bookkeeping in an _inode_, one per file, found by its number. The 1974 paper lists what an inode holds: the file’s owner, its protection bits, the disk addresses of its contents, its size, its time of last modification, its number of links, and whether it’s a directory. Linux’s [inode(7)](https://man7.org/linux/man-pages/man7/inode.7.html) manual page lists the same fields today, with a few more timestamps. When you run `stat` on a file, you’re reading its inode, and `ls -i` prints its number. One thing is missing from that list: the file’s name. ### Directories: tables of names A directory is a file whose contents are a table. Each entry holds a name and an inode number, and nothing else. The paper says so directly: “a file does not exist within a particular directory; the directory entry for a file consists merely of its name and a pointer to the information actually describing the file.” By convention, every directory also holds two entries of its own: `.`, which points to the directory itself, and `..`, which points to its parent. Two consequences follow, and the rest of this page leans on both. First, a file doesn’t know its path. Inode 87 in the following figure has no record that it’s called `app.ts`, or that it sits in `src/`. Second, renaming a file, or giving it a second name, changes only directory tables. The inode and the file’s bytes stay where they are. **A directory and an inode** The directory `src/`, inode 12, is a table of four entries: `.` points to 12, `..` to 31, `app.ts` to 87, and `util.ts` to 90. The entry for `app.ts` leads to inode 87, which holds the file’s type (regular file), mode `rw-r--r--`, owner `agent`, size of 9,210 bytes, link count of 1, modification time, and the numbers of its three blocks: 1904, 1905, and 7702. Those three 4 KiB blocks on the disk hold the bytes. The name `app.ts` appears only in the directory. ### Path lookup, one name at a time A path is a set of directions, not an address. To resolve `/home/agent/repo/src/app.ts`, the kernel starts at the process’s root directory and looks up `home` in its table. It checks that the result is a directory and that the process has search permission on it, then looks up `agent` there, and so on to the last name. A path without a leading `/` starts from the process’s working directory instead. (The rules in this section are from [path\_resolution(7)](https://man7.org/linux/man-pages/man7/path_resolution.7.html).) A symbolic link changes the route partway. Its contents are another path, and the kernel resolves that path starting from the directory that holds the link. Linux follows at most 40 symbolic links while resolving one path, and fails with `ELOOP` after that. Each step can fail, and each failure has its own error. A directory without search permission gives `EACCES`, a missing name gives `ENOENT`, and a regular file where a directory should be gives `ENOTDIR`. Step through a few paths in the following explorable, both as the user who owns the files and as one who doesn’t. **Resolving a path** An explorable that resolves one of four paths, one name at a time, as either the user `agent` or the user `guest`. `/home/agent/repo/src/app.ts` takes five lookups, from the root directory (inode 2) through `home` (11), `agent` (20), `repo` (31), and `src` (12) to the file, inode 87. The relative path `repo/node_modules/.bin/tsc` starts at the working directory, `/home/agent`, and reaches `tsc` in `.bin`, inode 47, which is a symbolic link containing `../typescript/bin/tsc`. The kernel resolves that path from `.bin`: `..` leads to `node_modules` (44), then `typescript` (48), `bin` (49), and the executable `tsc`, inode 50, after eight lookups in all. `/home/agent/repo/src/main.ts` fails with `ENOENT` because `src` has no entry by that name. `/home/agent/repo/build.sh/x` fails with `ENOTDIR` because `build.sh` is a regular file. As `guest`, every path stops with `EACCES` at `/home/agent`, whose mode `rwxr-x---` gives users outside its owner and group no search permission. A path with five names costs five lookups. Linux remembers recent results in the dentry cache, or _dcache_, which “caches information about names in each filesystem to make them quickly available for lookup”. For a local file system such as ext3, XFS, or Btrfs, the kernel’s documentation says the dcache is always accurate, though not always complete. A file system shared across a network can’t make that promise, because another machine can change a directory at any time. (From the kernel’s [Pathname lookup](https://www.kernel.org/doc/html/latest/filesystems/path-lookup.html) documentation.) So an NFS client sends the server a `LOOKUP` for each name it can’t trust from its cache, and a FUSE file system receives each such lookup as a request to its daemon. Where those requests go, and how long each takes, accounts for much of the speed difference between network file systems. For more information about what each request costs, see [Anatomy of a file create over a network](https://archil.com/guides/how-file-systems-work/anatomy-of-a-file-create). ### Descriptors and open file descriptions When the walk ends, `open()` has an inode, but it doesn’t hand the inode back. In the words of the [open(2)](https://man7.org/linux/man-pages/man2/open.2.html) manual page, “A call to open() creates a new open file description, an entry in the system-wide table of open files. The open file description records the file offset and the file status flags.” The process gets back a file descriptor, a small integer that refers to that description. So there are three layers. The descriptor belongs to one process. The open file description holds the offset and flags such as `O_APPEND`. The inode is the file. The middle layer explains two behaviors that surprise people. A child created by `fork()` inherits its parent’s descriptors, and they refer to the same descriptions, so parent and child share one offset. And every `open()` makes a new description, so two opens of the same file keep separate offsets. The descriptor also outlives the name. It “is unaffected if path is subsequently removed or modified to refer to a different file.” From here on, the path no longer matters. Reads and writes name a descriptor, never a path. **Descriptors, descriptions, and an inode** Process A opens `app.ts` read-only and gets descriptor 3, which points to open file description 1 with offset 0. A reads 4,096 bytes, and the offset becomes 4,096. A forks process B, whose descriptor 3 points to the same description. B reads 4,096 bytes, and the shared offset becomes 8,192, so B’s read moved A’s offset. A opens `app.ts` again with `O_APPEND` and gets descriptor 4, which points to a second description with its own offset, set to the end of the file before each write. Both descriptions point to inode 87. ### Reads and writes at byte offsets `read(fd, buf, n)` reads up to _n_ bytes from the description’s offset and moves the offset forward. `pread()` and `pwrite()` take an explicit offset instead. A write in the middle of a file replaces exactly the bytes it covers, and a write past the end makes the file longer. The 1974 paper already said, “There is no distinction between ‘random’ and sequential I/O.” `O_APPEND` adds one guarantee. Before each write, the kernel moves the offset to the end of the file, and “The adjustment of the file offset and the write operation are performed as an atomic step.” (From [write(2)](https://man7.org/linux/man-pages/man2/write.2.html), which is also the source of the two quotations that follow.) Two processes that append to one log don’t overwrite each other’s lines. A write lands in the _page cache_, the kernel’s in-memory copy of the file’s blocks, and reaches the disk later. A successful `write()` “does not make any guarantee that data has been committed to disk.” Only `fsync()` asks for that. For more information about what it promises, see [Where fsync returns from](https://archil.com/guides/how-file-systems-work/where-fsync-returns-from). What POSIX does promise readers is narrow: “a read(2) that can be proved to occur after a write() has returned will return the new data.” It promises nothing about a sequence of reads. A program that reads a large file in pieces can see some pieces from before a write and some from after. **A read that spans a write** A reader copies a 256 KiB file in four 64 KiB reads. After its second read, a writer overwrites the whole file with version 2. The reader ends up holding v1, v1, v2, v2, a mix that the file never contained at any moment. Every call did what POSIX says it should. This is the gap between a whole-file API and POSIX. `GetObject` returns one version of an object. A `readFile()` built on POSIX makes several `read()` calls, and nothing ties them to one version. Writes have the mirror-image problem: a program that crashes halfway through leaves half its writes behind. The standard way to replace a file all at once is to write a new file and rename it over the old one. ### Rename: one entry, one step The [rename(2)](https://man7.org/linux/man-pages/man2/rename.2.html) manual page makes the promise that git’s lock files rely on: “If newpath already exists, it will be atomically replaced, so that there is no point at which another process attempting to access newpath will find it missing.” In terms of the earlier figures, a rename removes one entry from a directory table and adds one, and every other process sees the swap as a single step. The page is candid about one detail: there may be a window in which both the old name and the new name refer to the file. Renaming a directory is the same operation. Renaming `build` to `dist` changes one entry in the parent directory and, if the directory moves to a new parent, its own `..` entry. The 10,000 files inside don’t move, because none of them records its path. A rename between two directories is the hardest call on this page to get right. Moving a directory into one of its own descendants would cut a loop out of the tree, so the kernel must check ancestry while nothing else rearranges it. Linux handles this with one lock per file system, `s_vfs_rename_mutex`, which every cross-directory rename takes. Its documentation notes that “the only operation that could introduce loops is cross-directory rename.” (From the kernel’s [Directory Locking](https://www.kernel.org/doc/html/latest/filesystems/directory-locking.html) documentation.) On one machine, that lock is a mutex. Across machines, something must play its part, and NFS gives the job to the server: its `RENAME` is one request, and the protocol says it “must be atomic to the client.” (From [RFC 1813](https://www.rfc-editor.org/rfc/rfc1813), the NFS version 3 specification, section 3.3.14.) An object store has no rename at all, so a client that wants one must copy each object to its new key and delete the old one. The following explorable runs `mv build dist` both ways and lets you choose when a second machine looks. **Rename versus copy** Machine A runs `mv build dist` on a directory of eight files, and machine B lists `build/` and `dist/` at a moment you choose. On a file system, the rename happens at one instant: before it, `build/` holds all eight files and `dist/` doesn’t exist, and after it the reverse is true. No listing shows both. On a bucket, the client copies each object to its new key and then deletes the old one, eight copies and eight deletes in all. A listing partway through shows both directories, each partly full, and between a file’s copy and its delete, that file appears in both. Each copy is atomic for its own object, but the directory as a whole never is. ### Links, and the last close A hard link is a second directory entry that holds the same inode number. The 1974 paper describes the whole operation: making a link “involves creating a directory entry with the new name, copying the i-number from the original file entry, and incrementing the link-count field of the i-node.” Neither name is the original, and both are equal. A hard link can’t cross file systems, because an inode number means something only within its own file system, and it can’t point to a directory. (See [link(2)](https://man7.org/linux/man-pages/man2/link.2.html) and [inode(7)](https://man7.org/linux/man-pages/man7/inode.7.html).) A symbolic link is a different thing: a small file whose contents are a path. The [symlink(7)](https://man7.org/linux/man-pages/man7/symlink.7.html) manual page calls it “a pointer to another name, and not to an underlying object.” A symlink can cross file systems and point to directories. It also breaks when its target name goes away, even if the file lives on under another name. Removing a file is `unlink()`. It deletes one name and decrements the inode’s link count, and the data goes when the count reaches zero, with one exception. The [unlink(2)](https://man7.org/linux/man-pages/man2/unlink.2.html) manual page puts it this way: “If the name was the last link to a file but any processes still have the file open, the file will remain in existence until the last file descriptor referring to it is closed.” That exception is hard for a network file system. The NFS version 3 specification says of it, “It is impossible for a stateless server to implement these semantics.” The workaround it suggests is the one NFS clients use: on remove, the client renames the file to a hidden name, and it deletes the file on the last close. (From [RFC 1813](https://www.rfc-editor.org/rfc/rfc1813), section 4.2.) **Names, links, and the last close** An explorable directory, `/data`, starts with one name, `model.bin`, for inode 77, a 100 MB file with a link count of 1. `ln model.bin current.bin` adds a second entry for inode 77 and raises its link count to 2, without copying any data. `ln -s model.bin latest` creates inode 78, a symbolic link whose contents are the name `model.bin`, and leaves inode 77’s link count alone. `exec 3< model.bin` opens inode 77 and keeps descriptor 3. Each `rm` removes one name and lowers the link count by one. After `rm model.bin`, `cat latest` fails with “No such file or directory”, even though inode 77 still exists under `current.bin`. Inode 77’s data is freed only when its link count is zero and no descriptor has it open, so with descriptor 3 still open, the data stays until `exec 3<&-` closes it. ### Permission bits An inode’s mode holds three groups of three bits, for the owner, the group, and everyone else. The kernel uses the first group if you own the file, the second if you belong to its group, and the third otherwise. In each group, the bits grant read, write, and execute. On a directory, the meanings shift to fit a table of names. Execute means _search_: permission to look a name up in the directory, which is the check that the path explorable failed as `guest`. Write permission on a directory is what lets a process add and remove its names, so `unlink()` fails with `EACCES` without it, whatever the mode of the file being removed. (See [chmod(2)](https://man7.org/linux/man-pages/man2/chmod.2.html) and [unlink(2)](https://man7.org/linux/man-pages/man2/unlink.2.html).) The sticky bit on a world-writable directory such as `/tmp` narrows that again: a file in it can be renamed or deleted only by its owner, the directory’s owner, or a privileged process. Git keeps only one of these bits: its index records each regular file as mode 0755 or 0644, and nothing finer. (From git’s [index format](https://git-scm.com/docs/index-format).) That one bit is still what decides whether `./build.sh` runs. ### An object store’s model S3 has no inodes, no directories, and no descriptors. AWS’s documentation says that general purpose buckets “have a flat structure instead of a hierarchy like you would see in a file system.” A key is a string that may contain slashes, and an object is a complete set of bytes stored under one key. (From [Organizing objects in the Amazon S3 console by using folders](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html).) The interface is small: - `PutObject` stores a whole object, replacing any object under that key. - `GetObject` reads an object, whole or as a byte range. - `ListObjectsV2` returns keys in lexicographic order, up to 1,000 per request, optionally only those that start with a prefix. - `CopyObject` copies an object, atomically for objects up to 5 GB, and `DeleteObject` removes one. An object’s user-defined metadata can’t change in place either. The documentation says that “The only way to modify this metadata is to make a copy of the object and set the metadata.” (From [Working with object metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html) and [Copying, moving, and renaming objects](https://docs.aws.amazon.com/AmazonS3/latest/userguide/copy-object.html).) What S3 does guarantee is strong consistency. Since December 2020, a read after a successful write returns the new object, and list operations are strongly consistent too. (From AWS’s announcement, [Amazon S3 now delivers strong read-after-write consistency](https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3-now-delivers-strong-read-after-write-consistency-automatically-for-all-applications/).) Directories are something a client infers from a listing. Called with the delimiter `/`, `ListObjectsV2` groups keys that share a prefix into `CommonPrefixes`, which, in the API reference’s words, “act like subdirectories in the directory specified by `Prefix`.” (From the [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) API reference.) A FUSE client draws its tree from those prefixes. An empty directory has no keys to imply it, so the S3 console makes one by storing a 0-byte object whose key ends in a slash. **Directories by implication** A bucket holds four keys: `README.md`, a 0-byte marker object `docs/`, `src/app.ts`, and `src/util.ts`. A client lists with the delimiter `/` and gets `README.md` plus the prefixes `docs/` and `src/`, and draws a tree from them. The client deletes `src/app.ts` and `src/util.ts`. A listing of the prefix `src/` now returns nothing, so the directory `src/` is gone, because no key says it exists. The empty `docs/` survives only because of its marker object. ### Where the mapping breaks The following table maps each POSIX call onto the nearest request that an S3 general purpose bucket offers. Where a client’s workaround is worth knowing, it names Mountpoint for Amazon S3, AWS’s open-source client. (Mountpoint’s behavior is from its [semantics documentation](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md): it sets one mode and owner for every file at mount time, with `--file-mode`, `--dir-mode`, `--uid`, and `--gid`, and it supports neither hard nor symbolic links.) | POSIX call | Nearest S3 request | What’s lost | | ------------------- | -------------------------------------------------------- | ----------------------------------------------------------------- | | Look up a name | `HeadObject`, or a listing of the prefix for a directory | A directory has no object of its own to find | | Read a directory | `ListObjectsV2` with a delimiter | A request per page of keys, one page after another | | Read at an offset | `GetObject` with a byte range | Nothing: ranged reads work well | | Write at an offset | `PutObject` of a new whole object | The write is as large as the file | | Append | None on general purpose buckets | The object is rewritten | | Rename a file | `CopyObject`, then `DeleteObject` | The pair isn’t atomic, and the data is copied | | Rename a directory | A copy and a delete for every key | Atomicity, and time proportional to the contents | | Hard link | None | The nearest thing is a copy that stops tracking the original | | Symbolic link | None | Each client invents a representation, if it supports links at all | | `chmod`, `chown` | User-defined metadata, changed by copying the object | Mountpoint fixes modes at mount time instead | | Unlink an open file | `DeleteObject` | A reader with the file open can lose it | Four of those rows need a closer look here. [A file system on an object store](https://archil.com/guides/how-file-systems-work/a-file-system-on-an-object-store) follows each of them further, with the requests they send and what they cost. #### No rename A client that offers rename over a bucket copies each key and deletes the old one. Each `CopyObject` is atomic for its own object, so every file arrives whole, but nothing makes the set atomic, which is the race in the rename explorable. AWS’s own console instructions show the same gap from the other side. They warn that when you delete a folder, you should wait for the delete to finish before adding objects to it, or “new objects might be deleted as well.” (From [Organizing objects in the Amazon S3 console by using folders](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html). S3 Express One Zone, a storage class in directory buckets, can rename single objects atomically, and Mountpoint uses that where it’s available.) #### Whole-object writes A write at an offset changes only the bytes it covers. On a bucket, the same change means a new object. Changing 4 KiB of a 1 GiB file writes 1 GiB, which is 262,144 times the change. A client can copy the unchanged ranges on the server side, as s3fs does with multipart copy, but it still produces a whole new object. (From the [s3fs README](https://github.com/s3fs-fuse/s3fs-fuse): “random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy”.) Appending a line to a log has the same cost, because general purpose buckets can’t append. #### Listing is a scan Reading a directory on a file system reads one table. On a bucket, it’s one `ListObjectsV2` request per page, and each request needs the continuation token from the one before, so the pages come one at a time. A directory of 100,000 files takes 100 requests in a row, every time a client lists it without a cache. #### Directories by implication A directory on a bucket is an agreement among the keys that share its prefix. It has no object of its own, unless a client stores a marker, so it has nowhere to keep a mode, an owner, or a modification time. It appears when its first key is written and disappears when its last key is deleted, as in the earlier figure. ### Who keeps the inodes Every design that offers the calls on this page keeps an inode table somewhere, and much of its behavior follows from where: - **A local disk or a network block volume** keeps it on the block device, and the file system in your own kernel reads and changes it. - **NFS** keeps it on the server. The client sends requests named for the calls in this page, such as `LOOKUP`, `RENAME`, `LINK`, and `SYMLINK`, and the server’s file system carries them out. (The procedures are listed in [RFC 1813](https://www.rfc-editor.org/rfc/rfc1813), section 3.3.) - **A metadata-database design**, such as JuiceFS, keeps it in a database such as Redis or PostgreSQL, and stores file contents in the bucket as numbered blocks rather than as files at their paths. (From JuiceFS’s [architecture documentation](https://juicefs.com/docs/community/architecture/), which says that “you cannot find the original file directly in the object storage.”) - **A FUSE client over a bucket** keeps it nowhere. It infers names from keys each time it lists, which is why its rename is a copy, and its links, where it has them, are a convention of its own. ### A worked example: Archil Archil keeps its inodes and directory tables on its own servers, and a mounted client resolves paths and changes names against them. For a disk synced to your own bucket, the bucket holds a projection of that tree: each file’s contents as an ordinary object at its path. What the bucket can’t express stays in Archil. Mode bits, ownership, and extended attributes are visible to mounted clients but not to a program that reads the bucket. A symbolic link becomes a small text file that holds its target, and a hard link becomes a full copy. A directory rename is atomic for every mounted client, but a program that reads the bucket directly can see it partly done while the change syncs. (See [Data sources](https://docs.archil.com/concepts/data-sources). Disks that Archil manages store data in our own format instead, and you read them through a mount or the disk’s S3 API.) **What your bucket sees** On a mounted Archil disk, `run.sh` has its mode and owner, `latest` is a symlink to `v2/`, and `model.bin` and `current.bin` are two names for the same 100 MB inode. In the bucket that the disk syncs to, `run.sh` is an object that holds its bytes, `latest` is a small text file that contains `v2/`, and the hard link is a second full copy, so 100 MB of data takes 200 MB of the bucket. Modes, owners, and extended attributes stay in Archil. For what these limits mean for agents that run git and package managers on shared storage, see [Storage for agents](https://archil.com/guides/storage-for-agents). ## 2. Where fsync returns from Source: https://archil.com/guides/how-file-systems-work/where-fsync-returns-from · Part of How file systems work · Facts checked 2026-11-04 **In this chapter:** How one write travels from a program’s memory to a drive’s flash and then to other machines, what `write()`, `fsync()`, `fdatasync()`, and `O_DIRECT` each wait for, and why replicated storage acknowledges where it does. By the end, you can name the layer any storage system acknowledges from, and the failures that choice survives. Between your program and permanent storage sit several places a write can rest: the program’s own memory, the kernel’s memory, a queue on its way to the drive, the drive’s cache, the flash itself, and then other machines. Each layer above the flash exists because the one below it is slower, and each can tell the layer above that a write is done. Durability is the question of which layer a call waits for. This page follows a single 4 KiB write down through one Linux machine, then out across several. The examples are Linux and ext4, because their behavior is documented in detail, but the same layers appear in every operating system and file system. ### The page cache A drive is slow next to memory, and programs write in small, scattered pieces. If every `write()` went to the drive before returning, a program writing a log line at a time would wait on the drive for every line. So the kernel keeps file data in memory, in the page cache, and serves both reads and writes from there. When your program calls `write()`, the kernel copies the bytes into a page of that cache, marks the page dirty, and returns. Dirty means the page holds data the drive doesn’t have yet. The cache is shared by every process on the machine, so a program that reads the file a moment later gets the new bytes, even though the drive has never seen them. **Writes land in memory** A program calls `write()` twice. Each write lands in the page cache, which marks the page dirty, and `write()` returns. Then the program calls `fsync()`: both writes go down to the disk, the pages become clean, and `fsync()` returns once the data is there. This explains a common mistake in durability testing. If you kill a process and then read its file, you read the page cache, which still has every write. Every kind of storage passes that test, so it tells you nothing. ### Writeback on the kernel’s schedule Dirty pages don’t stay in memory forever. Kernel flusher threads wake up periodically and write out pages that have been dirty for long enough. In the Linux source, the defaults are a wake-up every 5 seconds and an age of 30 seconds. (`dirty_writeback_interval = 5 * 100` and `dirty_expire_interval = 30 * 100`, both in centiseconds, in the kernel’s [mm/page-writeback.c](https://github.com/torvalds/linux/blob/master/mm/page-writeback.c). They’re tunable as `vm.dirty_writeback_centisecs` and `vm.dirty_expire_centisecs`, described in the [kernel’s vm sysctl documentation](https://docs.kernel.org/admin-guide/sysctl/vm.html).) **A write nobody syncs** A program dirties a page at 2 seconds and never calls `fsync()`. Flusher threads wake every 5 seconds and write pages that have been dirty for 30 seconds, so the page is written at the 35-second wake-up. A power cut at any time in between loses the write, although `write()` returned long before. Memory pressure also forces writeback. When dirty pages pass a share of available memory, `dirty_background_ratio`, the flusher threads start writing in the background. Past a second, larger share, `dirty_ratio`, a process that keeps writing has to write out dirty data itself, so its `write()` calls slow down. The kernel source sets these at 10% and 20%. Most of the time, this is enough. A laptop that crashes can lose up to about half a minute of unsynced writes, and the programs that care call `fsync()`. Your Mac asks you to eject an SD card for the same reason: ejecting flushes what’s still in memory before you pull the card. ### The calls that wait A program that needs a write to survive a power cut has to ask. The Linux manual page for `fsync()` says it flushes a file’s modified data “to the disk device (or other permanent storage device) so that all changed information can be retrieved even if the system crashes or is rebooted.” Two more sentences matter as much. The flush “includes writing through or flushing a disk cache if present,” and the call “blocks until the device reports that the transfer has completed.” (From the [fsync(2) manual page](https://man7.org/linux/man-pages/man2/fsync.2.html).) `fsync()` flushes the file’s metadata too: its size, its timestamps, and where its blocks are. `fdatasync()` skips metadata that a later read doesn’t need. The man page’s example is the modification time, which a read can do without, whereas a change in file size has to be flushed. On Linux, `fdatasync()` is PostgreSQL’s default way of syncing its log. (The `wal_sync_method` setting in [PostgreSQL’s write-ahead log settings](https://www.postgresql.org/docs/current/runtime-config-wal.html).) You can also make every write wait, by opening the file with a flag. With `O_SYNC`, each `write()` behaves as if `fsync()` followed it. With `O_DSYNC`, it behaves as if `fdatasync()` followed it. `O_DIRECT` is different, and often misread. It tells the kernel to skip the page cache and move data between the program’s memory and the drive directly. Databases that manage their own cache use it to avoid holding every page twice. But skipping the cache isn’t a durability promise. The `open()` man page says `O_DIRECT` alone only “makes an effort to transfer data synchronously,” and that you need `O_SYNC` with it to get the full guarantee. (From the [open(2) manual page](https://man7.org/linux/man-pages/man2/open.2.html).) An `O_DIRECT` write can return while the data sits in the drive’s own cache. Step a write through the stack in the following figure. Each choice shows where the call returns, and what a crash, a power cut, or a lost machine would leave behind at that moment. **One write, step by step** The figure steps a 4 KiB write, and the file’s modification time, through five layers of one machine: program memory, the page cache, the queue to the drive, the drive’s write cache, and flash. A plain `write()` returns once the data is in the page cache. The page reaches the drive about 30 seconds later, and flash whenever the drive gets to it. `fdatasync()` returns only after the kernel has sent the data and a cache flush, so the data is on flash, but the modification time stays in memory. `fsync()` does the same for the data and the modification time. An `O_DIRECT` write skips the page cache and returns once the drive reports completion, which can be while the data is still in the drive’s volatile cache. `O_DIRECT` with `O_DSYNC` returns once the data is on flash. Data in the page cache survives a program crash. Only data on flash, or in the cache of a drive with power-loss protection, survives a power cut. Nothing in this machine survives losing the machine. ### Metadata and the journal Creating a file changes more than its data. The file system allocates blocks for the data, writes an inode that records the file’s size and where its blocks are, and adds an entry to the parent directory that maps the name to the inode. Each lives in a different place on the drive. For more information about inodes and directory entries, see [From a path to an inode](https://archil.com/guides/how-file-systems-work/from-a-path-to-an-inode). The `fsync()` man page warns about this: syncing a file “does not necessarily ensure that the entry in the directory containing the file has also reached disk.” For that, you also call `fsync()` on the directory. The order of these writes matters as much as whether they happen. Suppose a crash lands after the inode is written but before the data. The file now claims blocks that hold whatever they held before, possibly another file’s old contents. Journaling exists to rule out that state. A journal is a small area of the drive written in order. The file system records a group of related changes there as a transaction and seals it with a commit record. After a crash, it replays sealed transactions and ignores unsealed ones, so each group of changes lands whole or not at all. ext4 offers three modes, and the default, ordered mode, journals only metadata. Its documentation says ordered mode groups the metadata for a change with its data blocks, and “when it’s time to write the new metadata out to disk, the associated data blocks are written first.” ext4 commits a transaction at least every 5 seconds by default. (From the [ext4 administrator’s guide](https://docs.kernel.org/admin-guide/ext4.html), which also describes `data=journal` and `data=writeback`.) **Ordered mode** In ext4’s ordered mode, the new data is written to its final place on disk first. Then the metadata, meaning the inode and the directory entry, is written to the journal. A commit record seals the transaction, and from then on, recovery after a crash applies it. Later, the metadata is copied to its home in the main area and the journal space is reused. A journal keeps the file system consistent. It doesn’t make your writes durable. That still takes `fsync()`, and programs that skip it learn the difference eventually. In 2009, users of the new ext4 found files full of zeros, or empty, after a crash. ext4 delays choosing blocks for new data, so a rename could commit before the data it pointed at had anywhere to go. Ted Ts’o, the ext4 maintainer, answered that “if you really care about making sure something is on disk, you have to use fsync or fdatasync.” ext4 still added a workaround for the common case. (From [“ext4 and data loss”](https://lwn.net/Articles/322823/) by Jonathan Corbet, LWN, March 11, 2009. The workaround is the `auto_da_alloc` mount option, on by default, which the ext4 guide introduces with “Many broken applications don’t use `fsync()` when replacing existing files.”) The safe way to replace a file’s contents takes two syncs: ```c int fd = open("config.tmp", O_WRONLY | O_CREAT | O_TRUNC, 0644); write(fd, buf, len); fsync(fd); // the new contents are on disk close(fd); rename("config.tmp", "config"); // swap the name atomically int dir = open(".", O_RDONLY | O_DIRECTORY); fsync(dir); // the rename is on disk too close(dir); ``` ### The drive’s own cache The drive is a computer too, with its own memory. Many drives accept a write into that memory and report it complete before it reaches the flash or the platter. The kernel’s documentation describes such drives as signaling “I/O completion to the operating system before data actually has hit the non-volatile storage.” (From [Explicit volatile write back cache control](https://docs.kernel.org/block/writeback_cache_control.html) in the kernel documentation.) The kernel has two tools for this. A flush, `REQ_PREFLUSH`, guarantees that every write the drive has already completed is on non-volatile storage. Force Unit Access, `REQ_FUA`, marks one write so that the drive reports completion only “after the data has been committed to non-volatile storage.” `fsync()` uses these, which is how it writes through the drive’s cache. Turn on power-loss protection in the stack figure to see the other way out. The arrangement depends on the drive telling the truth. SQLite’s documentation is blunt about it: “most consumer-grade mass storage devices lie about syncing,” and USB sticks are “especially pernicious liars.” (From [How To Corrupt An SQLite Database File](https://www.sqlite.org/howtocorrupt.html), section 3.1.) Some drives built for data centers take the other route: capacitors that hold enough charge, when power fails, for the drive to finish writing its cache. Samsung’s description of the feature says the capacitors give the drive time “for the data flush from the DRAM to the NAND to occur.” (From Samsung’s [power loss protection](https://download.semiconductor.samsung.com/resources/others/Samsung_SSD_845DC_05_Power_loss_protection_PLP.pdf) white paper for its 845DC data-center drives.) A drive with no volatile cache to lose needs no flush, and the kernel completes empty flush requests without sending them to the drive. ### When fsync fails So far, `fsync()` has either succeeded or been interrupted by a crash. It can also fail, and in 2018 the PostgreSQL project discovered what Linux did then. PostgreSQL writes data pages through the page cache and later calls `fsync()` on each file. If the call fails, the natural reaction is to retry. Craig Ringer reported to the pgsql-hackers mailing list on March 28, 2018, that this was unsafe. (The thread is [“PostgreSQL’s handling of fsync() errors is unsafe and risks data loss at least on XFS”](https://www.postgresql.org/message-id/CAMsr%2BYHh%2B5Oq4xziwwoEfhoTZgr07vdGG%2Bhu%3D1adXx59aTeaoQ%40mail.gmail.com).) When writeback fails, LWN explained, file systems typically respond by “discarding the data in the affected pages and marking them as being clean.” The error is reported once. A retried `fsync()` finds no dirty pages, has nothing to write, and returns success. (From [“PostgreSQL’s fsync() surprise”](https://lwn.net/Articles/752063/) by Jonathan Corbet, LWN, April 18, 2018.) **A retry that succeeds** A program writes, dirtying a page. Writeback to the disk fails, and the kernel marks the page clean anyway, although the data isn’t on disk. The next `fsync()` returns the error `EIO`. The program retries, and the second `fsync()` returns 0, meaning success, although the data never reached the disk and the page that held it can be evicted. The kernel developers had reasons. Ts’o pointed out that the most common cause of I/O errors is a user pulling out a USB drive, and keeping the dirty pages of a vanished drive could fill memory. Linux 4.13 and later releases changed how writeback errors are tracked, so they’re harder to miss. Each error is still reported once, and the pages are still gone. (From PostgreSQL’s [Fsync Errors](https://wiki.postgresql.org/wiki/Fsync_Errors) wiki page, which lists the behavior of each kernel version.) PostgreSQL’s answer was to stop trusting the page cache after a failure. It now stops with a `PANIC` when `fsync()` fails, then recovers by replaying its write-ahead log, which still holds every committed change. The project made the change in version 12 and back-patched it to 9.4 through 11. The episode became known as “fsyncgate.” For a storage design, it means that after a failed sync, the page cache can’t be trusted, and recovery needs a copy written somewhere else. ### The machine as a failure domain Everything so far survives a power cut, at best. None of it survives losing the machine, because every layer, flash included, is inside it. In the cloud, losing the machine is routine. AWS’s documentation for EC2 instance store, the drives physically attached to the host, lists what survives. A reboot and a power failure keep the data. A stop, a termination, a retirement after the host fails, or a failed disk lose it. (From [Data persistence for Amazon EC2 instance store volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-store-lifetime.html).) A program can call `fsync()` perfectly and still lose everything it wrote there. Storage services publish durability as a probability, and the units differ. AWS designs a gp3 EBS volume for 99.8% to 99.9% durability, which it states as a 0.1% to 0.2% annual failure rate, for the whole volume. It designs S3 for 99.999999999% durability of each object over a year. (From the AWS pages on [EBS volume types](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volume-types.html) and [S3 data protection](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html).) Both get there the same way: by putting copies on several machines before saying a write is done. ### Replicate, then acknowledge To survive the loss of a machine, a write has to exist somewhere else before anyone is told it’s safe. The simplest form is a primary that forwards each change to a standby and waits for an answer. The design question is what the standby must do before it answers. PostgreSQL makes the choice explicit in its `synchronous_commit` setting. With `off`, a commit returns before even the primary’s log is flushed. With `local`, it waits for the primary’s own flush “but not for replication.” With `remote_write`, it also waits until the standby has “written it to their file systems,” which means the standby’s page cache. With `on`, the default, it waits until the standby has “flushed it to durable storage.” With `remote_apply`, it also waits until the standby has applied the change, so queries there see it. (Quoted from `synchronous_commit` in [PostgreSQL’s write-ahead log settings](https://www.postgresql.org/docs/current/runtime-config-wal.html). The standby levels apply when `synchronous_standby_names` names a standby.) Each step down the ladder adds a wait, and each protects against one more failure. Where the standby runs matters as much. A standby in the same zone shares that zone’s power and network, so one event can take both machines. **Who acknowledges** A primary and one standby. Acknowledging from the primary’s memory survives none of the five failures shown, not even a crash of the database process. Acknowledging from the primary’s disk survives a process crash and a power cut, and loses the commit if the primary machine is lost. Acknowledging once the standby has the write in memory survives losing the primary, but if the standby is in the same zone, a zone-wide power cut wipes its memory. Acknowledging once the standby has it on disk survives all five failures with the standby in another zone, and everything except losing the whole zone with the standby in the same zone. Acknowledging after the standby applies the change costs one more step and adds visibility on the standby, not durability. Amazon’s Dynamo paper shows a system choosing a middle rung on purpose. Its storage nodes buffer writes in memory and write them to disk in the background. To limit the risk, the coordinator chooses “one out of the N replicas to perform a ‘durable write’,” so at least one copy reaches disk before the write counts. (From [“Dynamo: Amazon’s Highly Available Key-value Store”](https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html), DeCandia and others, SOSP 2007.) ### Quorums Waiting for every replica has a cost: the slowest replica sets the pace, and a dead one stops all writes. A quorum waits for enough replicas instead of all of them. Raft, a consensus algorithm, commits a log entry “once the leader that created the entry has replicated it on a majority of the servers.” Its followers update their logs “on stable storage before responding,” so a reply means a copy on disk. A cluster of five servers keeps working through any two failures. (From Diego Ongaro and John Ousterhout, [“In Search of an Understandable Consensus Algorithm”](https://raft.github.io/raft.pdf), USENIX ATC 2014.) Dynamo generalizes the idea: with N replicas, waiting for W of them on a write and R on a read, “setting R and W such that R + W > N yields a quorum-like system,” because every read then overlaps at least one replica that took the latest write. **A majority of five** The leader appends entry 7 to its log and sends it to four followers. One follower is unreachable and one is slow. Two followers store the entry on disk and reply, so three of the five servers have it, a majority. The leader commits and answers the client. The slow follower stores the entry later. A quorum only helps when its members fail independently. Three replicas in one rack share a power supply and a switch. Three in one building share its power and its fire. Cloud providers draw this boundary as the availability zone: S3 stores objects “across a minimum of three Availability Zones,” which are “physically separated by a meaningful distance, many kilometers, from any other.” PlanetScale Metal runs databases on local NVMe and still makes sure “every write has reached stable storage in two availability zones before it’s acknowledged.” (From [S3 data protection](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html) and the FAQ on [PlanetScale Metal](https://planetscale.com/metal).) Crossing zones costs a longer round trip on every acknowledged write. That cost buys survival of the failures a single zone can’t protect against. ### Write-ahead logs Replication and crash recovery both get cheaper with one more idea. Instead of updating a file’s pages in place and syncing each one, a system appends a description of the change to a log, syncs the log, and acknowledges. It updates the real data structures later, in bulk. PostgreSQL’s documentation states the rule: changes to data files “must be written only after those changes have been logged.” Because of it, only the log “needs to be flushed to disk to guarantee that a transaction is committed,” and the log “is written sequentially,” which costs less than flushing scattered pages. (From [PostgreSQL’s introduction to write-ahead logging](https://www.postgresql.org/docs/current/wal-intro.html).) After a crash, replaying the log redoes whatever hadn’t reached the data files. **Log first, data later** Four changes to a balance are appended to the log, and each commit is acknowledged once its record is durable. The data files still say 100 when the machine crashes. On restart, replay applies the four records, the data files say 160, and the log up to the fourth record can be trimmed. The log is also the natural thing to replicate. PostgreSQL streams its log to standbys, and Raft is an algorithm for agreeing on a log. A storage system can keep recent writes in a replicated log and fold them into slower storage, such as an object store, long after they were acknowledged. ### The layers at a glance The following table lists each place a write can be acknowledged from, and which failures a write acknowledged there survives. | Acknowledged from | Program crash | Power cut | Machine lost | Zone lost | | ----------------------------------------------- | ------------- | --------- | ------------ | ---------------------------- | | Program memory (a buffered library call) | No | No | No | No | | Page cache (`write()`) | Yes | No | No | No | | Drive’s volatile cache (`O_DIRECT`) | Yes | No | No | No | | Drive’s flash, or a protected cache (`fsync()`) | Yes | Yes | No | No | | Another machine’s memory | Yes | Yes | Yes | Only if it’s in another zone | | Another machine’s disk | Yes | Yes | Yes | Only if it’s in another zone | | A majority of disks across zones | Yes | Yes | Yes | Yes | The two rows for another machine look alike, but they differ when that machine loses power too. A write held in its memory is gone, and a write on its disk comes back. ### A worked example: Archil Archil combines these pieces, and its design maps onto the layers on this page. Our client runs on the same machine as your program and keeps recent writes in memory, as the page cache does, so `write()` doesn’t wait for the network. The client sends changes to our servers in the background, and immediately when you call `fsync()`. The servers append those changes to a write-ahead log and replicate it across availability zones before `fsync()` returns. In the terms of the “Who acknowledges” figure, Archil acknowledges once other machines have stored the write, and those machines are in other zones. Archil is designed for 99.999% durability for writes in that log before they reach S3. Later, usually tens of seconds to minutes after a write, the log is folded into the disk’s object storage. (From [Data durability](https://docs.archil.com/details/architecture#data-durability) and [Consistency](https://docs.archil.com/details/consistency) in Archil’s documentation.) The page-cache rule applies to our client too. A write that hasn’t been synced exists only in the client’s memory, and a power cut loses it. Use `archil unmount`, which flushes before it detaches, rather than a plain `umount`, which doesn’t. For what this means for agents whose machines are deleted when a task ends, see [Storage for agents](https://archil.com/guides/storage-for-agents). ## 3. Anatomy of a file create over a network Source: https://archil.com/guides/how-file-systems-work/anatomy-of-a-file-create · Part of How file systems work · Facts checked 2026-11-04 **In this chapter:** What a program asks for when it creates one small file, why a network file system has to ask a server before it can answer, and how callbacks, leases, and delegations let a client answer from its own memory once it holds the right to. Creating a file looks like one action, and on a laptop it costs about as much as one. Over a network it can cost several round trips, and a job that creates thousands of files pays that cost thousands of times. This page builds up why, one step at a time. It starts with a create on a single machine, sends the same create to a server, and then follows the three ideas that let a client stop asking, which are callbacks, leases, and delegations. ### One file on one machine A program that unpacks a package creates each file with a handful of system calls: ```c fd = open("node_modules/react/index.js", O_CREAT | O_WRONLY, 0644); write(fd, buf, 4096); fchmod(fd, 0644); close(fd); ``` Each call changes a different part of the file system. `open` with `O_CREAT` adds a name to a directory and allocates an _inode_, the record that holds a file’s owner, mode, size, and timestamps. For more information about inodes and directories, see [From a path to an inode](https://archil.com/guides/how-file-systems-work/from-a-path-to-an-inode). `write` puts 4 KiB of data into the file. `fchmod` changes the mode in the inode. `close` releases the descriptor. On Linux, none of these calls waits for the disk. The kernel keeps directory entries, inodes, and file data in memory: the dentry cache, the inode cache, and the page cache. Each call changes memory and returns. The disk catches up later. ext4, for example, commits its journal every 5 seconds by default, and data reaches the disk when the kernel writes back dirty pages, or when a program calls `fsync()`. [Where fsync returns from](https://archil.com/guides/how-file-systems-work/where-fsync-returns-from) follows that path in detail. **A create on a local disk** A program creates a file with four calls: open with O\_CREAT, write 4 KiB, fchmod, and close. Each call changes something in the kernel’s memory, a directory entry, a cached page, or an inode’s mode, and returns at once. The disk is written later, when ext4 commits its journal, every 5 seconds by default. This works because one kernel owns this file system, so it’s the only place that can change a name in that directory. It never has to ask anyone whether `index.js` already exists. ### The same create, sent to a server Sun’s Network File System puts a file system behind a server, to give many machines “transparent remote access to shared files across networks.” The specification of its second version, published as [RFC 1094](https://www.rfc-editor.org/rfc/rfc1094) in March 1989, states a goal that shaped everything after it: “The NFS protocol was intended to be as stateless as possible.” A stateless server keeps no record of its clients. If it crashes and restarts, clients retry their requests and carry on, with no state to rebuild. The NFS client in the kernel turns each system call into a remote procedure call. NFS version 3, specified in [RFC 1813](https://www.rfc-editor.org/rfc/rfc1813), has a procedure for each kind of change, and the small file in the preceding section needs four of them: - `CREATE` makes the name and the inode. - `SETATTR` sets the mode that `fchmod` asked for. - `WRITE` sends the 4 KiB of data. NFSv3 lets the server hold this write in memory, marked `UNSTABLE`. - `COMMIT` asks the server to put that data on stable storage. Clients send the pending write and the commit when the file is closed. RFC 7530 makes that a rule for NFSv4 and notes that many NFSv3 clients already follow it. `CREATE` and `COMMIT` are on RFC 1813’s list of synchronous procedures: the server writes the change to stable storage before it replies. So each round trip includes a durable write on the server, as well as the network. The cost of a file is now the number of calls that wait, multiplied by the distance to the server. The following explorable does that multiplication. Move the server closer or farther away, and choose a job. **Round trips per file** Creating one small file takes four calls. Over NFS, three of them wait for the server: CREATE when the file is opened, SETATTR when its mode is set, and WRITE and COMMIT when it’s closed, for four round trips in all. A client that holds the directory answers all four itself. At 0.5 ms per round trip, an `npm ci` of 18,602 files waits for 18,602 × 4 × 0.5 ms, about 37 seconds. At 40 ms, it waits about 50 minutes. Real clients overlap some calls, so these totals show the size of the problem, not a prediction. Real clients overlap some of this work. A package manager runs several unpacks at once, and the kernel can have several requests in flight. But each file’s own calls happen in order, and a job of thousands of files still pays thousands of waits. ### Why only the server can decide Suppose two machines, A and B, both run `touch hello.txt` in the same shared directory at the same moment. Only one of them can create the file. A’s kernel doesn’t know about B’s request, and B’s kernel doesn’t know about A’s. The server is the only place where both requests arrive, so it’s the only place that can decide. NFS gives the server the tools to decide. A `CREATE` request can be `GUARDED`, which tells the server to check for a file with the same name first and fail with `NFS3ERR_EXIST` if one is there. It can also be `EXCLUSIVE`, which handles a retried request correctly by storing a verifier with the new file. RFC 1813 adds that after an exclusive create, the client “must issue a SETATTR to set the correct file attributes,” which is one reason `SETATTR` follows `CREATE` so often. **Who gets hello.txt?** Machines A and B both create hello.txt. Only the NFS server can decide who wins, so both ask. A wins, and B is told the file exists. Then A creates four more files that nobody else wants, and each one still makes a round trip, because the server has no way to know that nobody else is looking. The second half of that story is the expensive half. A goes on to create files that no other machine will ever touch, and every create still crosses the network. A stateless server keeps no record of who is working in which directory, so it can’t tell A that it has the directory to itself. A can’t know that either. Asking is the only safe choice, so A asks every time. ### What a client may cache A client can’t decide a create alone, but it can remember what it has read. NFS clients cache file attributes and directory contents, and treat them as current for a while. Linux’s defaults, from the `nfs(5)` man page, are 3 to 60 seconds for a file’s attributes and 30 to 60 seconds for a directory’s. Caching raises the question of when one client sees another’s change. NFS clients answer it with _close-to-open consistency_. When a program closes a file, the client sends its pending writes to the server. When a program opens a file, the client asks the server for the file’s current attributes, “regardless of the freshness of the file’s cached attributes,” in the man page’s words. If A closes a file before B opens it, B sees A’s changes. **Close-to-open, and a cached listing** Client A writes notes.txt and closes it, which sends WRITE and COMMIT to the server. Client B opens the file. The open always sends GETATTR, so B reads A’s new version. Then A creates new\.txt. B lists the directory, and its client answers from cached directory attributes, so new\.txt is missing. Linux trusts those attributes for at least 30 seconds by default. This is a good fit for files that one machine writes and another reads later, which is most sharing. It doesn’t make anything faster for a create: the close still waits for the server, and the next open still asks. It also leaves a window in which a listing is stale. The Andrew File System paper described NFS as a system “in which a file created on a workstation may not be visible on another workstation for 30 seconds.” ### Callbacks: the Andrew File System The Andrew File System was built at Carnegie Mellon, which planned for it to span more than 5,000 workstations. John Howard and six colleagues described its first two versions in “[Scale and Performance in a Distributed File System](https://www.cs.cmu.edu/~coda/absdir/s11-abstract.html),” published in _ACM Transactions on Computer Systems_ in February 1988. The paper is a careful record of a design that didn’t scale, the measurements that showed why, and the change that fixed it. AFS caches whole files on each workstation’s local disk. A process called Venus manages the cache and talks to the servers, called Vice. In the prototype, Venus treated every cached file as suspect: “Before using a cached file, Venus would verify its timestamp with that on the server responsible for the file. Each open of a file thus resulted in at least one interaction with a server, even if the file were already in the cache and up to date.” Programs that called `stat` before opening a file made it worse, because each `stat` checked the cache too. Two calls made up nearly 90% of the traffic to the servers: `TestAuth`, which validated a cache entry, and `GetFileStat`, which fetched the status of a file absent from the cache. Only 6% of calls moved file data. Performance was usually acceptable up to about 20 active users per server. In the team’s benchmark, `TestAuth` response times rose rapidly once more than five workstations ran it at the same time. The revised AFS turned the question around. In the paper’s words, “When a workstation caches a file or directory, the server promises to notify it before allowing a modification by any other workstation. This promise, called a Callback, dramatically reduces the number of cache validation requests received by servers.” Venus now assumes a cached entry is valid unless told otherwise, so an open of a cached file needs no message at all. **Checking every time, or being told** In the AFS prototype, Venus checks with the server on every open of a cached file, so six opens make six TestAuth calls that get the same answer each time. In the revised AFS, the first open fetches the file and the server records a callback, a promise to notify Venus before any other workstation changes it. The next four opens stay local. When another workstation stores the file, the server breaks the callback, Venus drops its copy, and the sixth open fetches the file again: two calls in all. The numbers moved a long way. The prototype ran the benchmark 70% slower than a workstation with a local disk, and the revised system ran it 19% slower. In the prototype, the benchmark took more than four times as long at a load of 10 as at a load of 1. In the revised system it took 36% longer at that load, and less than twice as long at a load of 20. The authors concluded that their goal of 50 users per server had been met. Callbacks also let workstations resolve path names from their caches, because a cached directory with a callback on it can be trusted. Callbacks cost something too. The server now keeps state for every cached file, and must notify every holder before it allows a change. A rebooted workstation can’t trust anything in its cache and has to revalidate each entry the first time it’s used. And callbacks covered reading, not writing names: “For reasons of integrity, modifications to a directory are made directly on the server responsible for that directory.” In AFS, a create still went to the server. ### Leases: a promise with an expiry A callback is a promise with no end date, which is a problem when machines fail. If the server can’t reach a workstation to break a callback, it can wait forever, or go ahead and let the workstation read a stale copy. In 1989, Cary Gray and David Cheriton at Stanford proposed a way out in “[Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency](https://dl.acm.org/doi/10.1145/74850.74870),” published in the proceedings of the 12th ACM Symposium on Operating Systems Principles. Their definition is short: “A lease is a contract that gives its holder specified rights over property for a limited period of time.” A cache that holds a lease on a file can serve reads of it without asking the server. When another client wants to write, the server must get the leaseholder’s approval, or wait until the lease has expired. A lease can cover the file’s name and permissions, not only its contents, so a rename counts as a write. The expiry is what makes leases tolerate failure. If a leaseholder crashes or the network splits, the server waits out the lease and then lets the write proceed, knowing nobody can still be relying on the old copy. A failure costs the writer time, and no client reads a stale copy. A server that restarts only has to remember the longest lease it granted, and hold writes for that long. **Approval, or the end of the term** Client A reads a file and gets a lease with it, so its next reads need no message. Client B writes the file. The server asks A to approve, A drops its copy and approves, and the write is applied. Then A takes a new 10-second lease and crashes. B writes again. The server gets no answer from A, so it waits until A’s lease term is over and then applies the write. The write is late, but no client ever reads a stale copy. The paper then asks how long a lease should last, and the answer is shorter than you might expect. In their model, built on file access data from Stanford’s V system, a 10-second term on a file cached by one client cut the traffic spent on consistency to 10% of what a zero-length term needed. Longer terms added little. Gray and Cheriton placed AFS on the same scale: its prototype amounted to a lease term of zero, and its revised version to a term of infinity. Short terms also limit _false sharing_, where a writer has to wait for a lease that’s held by a client no longer using the file. ### Delegations in NFSv4 NFS version 4 gave up statelessness. Its specification, now [RFC 7530](https://www.rfc-editor.org/rfc/rfc7530), keeps a client’s opens, locks, and other state on the server under a single lease that the client renews. On that foundation it adds _delegations_, a mechanism “that allows many decisions normally made by the server to be made locally by clients.” An open delegation hands a client the right to handle opens and closes of one file by itself. With a read delegation, other clients may not change the file. With a write delegation, the client may change the file’s data, and only the attributes tied to it: size, modification time, and the change attribute. While it holds either, the client “does not send OPENs or CLOSEs to the server.” When another client wants the file, the server recalls the delegation with a callback, as AFS did. The holder sends its modified data back and returns the delegation, and the second client’s request waits until it does. NFSv4.1, now [RFC 8881](https://www.rfc-editor.org/rfc/rfc8881), added _directory delegations_ for exactly the case on this page. The RFC names compilation as a workload with many lookups of files that don’t exist yet, and credits AFS and DFS with showing that state about directory contents “can greatly reduce network traffic.” Directory delegations are optional for servers, and they stop short of writing: “The delegation is read-only and the client may not make changes to the directory other than by performing NFSv4.1 operations that modify the directory or the associated file attributes so that the server has knowledge of these changes.” A client with a directory delegation can list the directory and miss on lookups from memory. It still sends every create to the server. The following table summarizes what each mechanism lets a client do without asking the server. | Mechanism | Reuse cached data and names | Open and close a file | Write a file’s data | Create a name in a directory | | ---------------------------- | --------------------------------- | --------------------------------- | --------------------------- | ---------------------------- | | NFSv3 attribute cache | For a few seconds, then ask again | No | No, flushed on close | No | | AFS callback | Until the server breaks it | Opens, yes | Yes, stored on close | No | | Lease | Until the term ends | Yes, if the lease covers the name | No, writes go to the server | No, writes go to the server | | NFSv4 write delegation | Yes | Yes | Yes | No | | NFSv4.1 directory delegation | Yes, for the directory | No | No | No | | Holding the directory | Yes | Yes | Yes | Yes | ### Answering a create from memory The last row is what a small-file job needs. For a client to answer `open(O_CREAT)` by itself, the server has to hand it the directory: the exclusive right to add and remove names there. Everything else follows the patterns this page has already built. - The right has to be exclusive, like a write delegation, because two clients deciding names in one directory is the conflict the server exists to prevent. - The server needs a way to take it back. When another client asks for the directory, the server recalls it, the holder sends every change it made, and only then does the server answer the second client. - The changes still travel, but not one at a time. A client that created 500 files sends them to the server together, so 500 waits become one. - Durability is a separate question. A create that exists only in the holder’s memory is not yet safe, exactly as a write on a local disk isn’t safe before `fsync()`. For what has to happen before `fsync()` returns, see [Where fsync returns from](https://archil.com/guides/how-file-systems-work/where-fsync-returns-from). Try it in the following explorable. Create a few files from A while nobody holds the directory, then hand A the directory and create a few more. Then create a file from B. **Who decides a create?** Two clients create files in one shared directory. When nobody holds the directory, every create from either client is one round trip, and the server decides each one. A second create of hello.txt fails because the name exists. When client A holds the directory, A decides its own creates in memory with no round trips, and its new names exist only in A’s memory until they’re sent. When B then creates a file, the server recalls the directory from A, A sends all of its creates in one message and gives the directory back, and the server answers B. B waits for two round trips: its own and the recall. The explorable also shows the cost. B’s first create after A took the directory waited for a recall as well as its own round trip. If A and B take turns creating files in the same directory, every turn is a recall, and that’s slower than if nobody had held the directory at all. Gray and Cheriton’s advice for leases applies here too: when another client is about to change the data, the cheapest grant is none. Holding a directory pays off when one client does most of the work in it, which is how a clone, an install, or a build behaves. For how other clients see the holder’s changes, see the next chapter, [One file, many caches](https://archil.com/guides/how-file-systems-work/one-file-many-caches). ### What still costs a round trip Holding a directory removes the waits on what a client creates and changes. Three kinds of wait remain, whatever the design. - **Reading something for the first time.** A client can answer from memory only what it has already fetched. A large tree that a job has never read costs a round trip per miss, unless the client fetches it ahead of time. - **Making changes durable.** `fsync()` has to reach storage that will outlive the machine, so it waits for at least one round trip to wherever that storage is. - **Taking the directory.** The first client to hold a directory asks for it once, and a client that wants a directory someone else holds waits for a recall. These are the waits to count when you read a vendor’s description of its protocol. Ask what happens on a create, an open, and a close, and which of them wait for a reply. ### A worked example: Archil’s checkout Archil, the file system we build, uses this approach. A client that holds part of the file system is said to have it _checked out_. Our docs and CLI call the grant a delegation, and the commands are `archil checkout` and `archil checkin`. (The modes and commands in this section are from [Sharing disks](https://docs.archil.com/concepts/sharing-disks) in Archil’s documentation.) A client mounts a disk in one of three modes: - In the default exclusive mode, the client that mounts a disk holds all of it. It answers creates, writes, renames, and attribute changes from memory, and sends the changes to our servers in the background and on every `fsync()`. Other machines can mount the disk read-only. - In shared mode, a client can read the whole disk, and it writes under paths it has checked out with `archil checkout `. Creating a file in a directory nobody holds checks that file out automatically. - In conditional mode, any number of clients write the same paths at once. Creates, writes, renames, and deletes run on our servers before they return, as they do over NFS, the last write wins, and the docs say performance falls to that of a remote file system. A client that does most of the writing in a path can still check it out, to write there at local speed while other clients read. A checkout differs from a lease in one way that matters. It doesn’t expire. If a client crashes while holding a path, nobody can write there until someone runs `archil checkout --force`, and that discards the crashed client’s writes that hadn’t reached our servers. Gray and Cheriton’s lease takes the other side of that trade: it waits out the term and loses nothing, at the cost of a delay after every failure. Writes that `fsync()` returned for are already on our servers, so a forced checkout loses at most what the program hadn’t synced. For what these round trips cost agents that install packages and clone repositories, see [Storage for agents](https://archil.com/guides/storage-for-agents). ## 4. One file, many caches Source: https://archil.com/guides/how-file-systems-work/one-file-many-caches · Part of How file systems work · Facts checked 2026-11-04 **In this chapter:** What a second machine sees after you write a file, and who gets to write it next. By the end you can tell linearizable, close-to-open, and eventual consistency apart, explain how a cache learns it’s stale, and say why leases exist and why locks across machines are harder than they look. On one computer, sharing a file is a solved problem that nobody notices. Two programs open the same file, one writes, the other reads, and the reader sees the write. Put the two programs on different computers, and every part of that sentence needs a decision: where the bytes live, who keeps a copy, when a copy goes out of date, and who may write next. This page builds those decisions up one at a time, starting from a single machine. Each step is a problem that some system’s designers had to solve, and the examples are their systems: AFS at Carnegie Mellon, NFS at Sun, Amazon S3, and Google’s Chubby lock service. ### One copy, then many On one machine, every process that reads a file goes through the kernel’s page cache, so there’s exactly one cached copy of each page. A write lands in that copy, and the next read, from any process, finds it there. POSIX writes this down as a rule: “If a `read()` of file data can be proven (by any means) to occur after a `write()` of the data, it must reflect that `write()`, even if the calls are made by different threads.” The same rationale adds that the rule “is particularly significant for networked file systems, where some caching schemes violate these semantics.” (From the rationale of [`write()` in POSIX.1-2024](https://pubs.opengroup.org/onlinepubs/9799919799/functions/write.html), The Open Group Base Specifications Issue 8.) **One copy, then many** On one machine, process 1 writes version 2 of `plan.md` into the kernel’s single page cache, and process 2’s next read returns version 2. On two machines, each has its own cache. Machine A writes version 2 into its cache, and machine B reads version 1 from its own. After A’s write reaches the server, B’s copy stays stale until B asks the server again. Give a second machine its own cache, and that rule stops coming for free. Each machine now holds a copy, and a copy can be wrong. Cary Gray and David Cheriton, writing about distributed file caches in 1989, defined what “consistent” should mean: “the behavior is equivalent to there being only a single (uncached) copy of the data except for the performance benefit of the cache.” (Gray and Cheriton, [“Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency”](https://www.andrew.cmu.edu/course/15-440/assets/READINGS/gray1989.pdf), SOSP 1989, section 1.) Everything that follows is either a way to get close to that single copy, or a decision to settle for less in exchange for fewer round trips to a server. ### Writes are bytes at offsets Before asking when a write becomes visible, be precise about what a write is. In object storage, a `PUT` replaces a whole object. Two clients that `PUT` the same key leave one complete version or the other. A file system has no such operation. `write()` puts some bytes at some offset, and a program saving a large file makes many of those calls. POSIX promises that each call is atomic, and nothing more about writers who overlap: it “does not specify the behavior of concurrent writes to a regular file from multiple threads, except that each write is atomic”, and “applications should use some form of concurrency control.” So when two programs save the same file at the same moment, the last writer wins each range of bytes separately. **Last writer wins, one write at a time** Two agents save their own version of `notes.md` to shared storage at the same moment, each as six 64 KiB writes. The writes arrive interleaved, and the last write to each piece stays, so the file ends up B, B, A, A, B, A. Every write succeeded, and neither agent saw an error, but the file matches neither version. Appending doesn’t escape this across machines. Linux’s `open(2)` man page warns that `O_APPEND` “may lead to corrupted files on NFS filesystems if more than one process appends data to a file at once”, because NFS has no append operation, so the client has to simulate one, “which can’t be done without a race condition.” (From the description of `O_APPEND` in the [open(2) man page](https://man7.org/linux/man-pages/man2/open.2.html).) Keep this in mind through the next section. A consistency model decides _when_ other machines see a write. None of them decides which of two conflicting writes should survive. That takes coordination: leases, ownership, and locks, later on this page. ### Three promises on one timeline Storage systems describe their consistency with a handful of named promises. They differ in one thing: how long after a write a reader on another machine may still see the old data. **Linearizability** is the strongest. Maurice Herlihy and Jeannette Wing defined it in 1990 as “the illusion that each operation applied by concurrent processes takes effect instantaneously at some point between its invocation and its response.” (Herlihy and Wing, [“Linearizability: A Correctness Condition for Concurrent Objects”](https://cs.brown.edu/~mph/HerlihyW90/p463-herlihy.pdf), ACM Transactions on Programming Languages and Systems 12(3), July 1990, abstract.) For a file, it means that once A’s `write()` returns, any read that starts afterward, on any machine, returns the new bytes. That is the POSIX rule stretched across a network. Its price is that before A’s write returns, every cached copy must be updated or thrown away, or every read must ask the server. **Close-to-open** consistency comes from the Andrew File System (AFS). Its designers at Carnegie Mellon wrote down their rules in 1988: “Writes to an open file by a process on a workstation are visible to all other processes on the workstation immediately but are invisible elsewhere in the network.” And: “Once a file is closed, the changes made to it are visible to new opens anywhere on the network. Already open instances of the file do not reflect these changes.” (Howard and others, [“Scale and Performance in a Distributed File System”](https://www.andrew.cmu.edu/course/15-440/assets/READINGS/howard1988-tocs.pdf), ACM Transactions on Computer Systems 6(1), February 1988, section 3.) The Linux NFS client keeps the same promise. When a program opens a file, the client checks with the server, and when the program closes it, the client “writes back any pending changes”. (From “Close-to-open cache consistency” in the [nfs(5) man page](https://man7.org/linux/man-pages/man5/nfs.5.html).) The promise covers an open that follows a close, and nothing else. A reader that already has the file open, or a writer that hasn’t closed it, is outside it. **Cached for a fixed time** is the promise most clients actually keep for metadata. A client that keeps each entry for _T_ seconds before asking again guarantees that what it shows is at most _T_ seconds older than the server’s copy. The Linux NFS client caches a file’s attributes for 3 s to 60 s and a directory’s for 30 s to 60 s by default, set by the `acregmin`, `acregmax`, `acdirmin`, and `acdirmax` mount options. **Eventual consistency** is the weakest useful promise. Werner Vogels put it as “if no new updates are made to the object, eventually all accesses will return the last updated value”, and called the gap before then the inconsistency window. (Vogels, [“Eventually Consistent”](https://www.allthingsdistributed.com/2008/12/eventually_consistent.html), revised for ACM Queue, December 2008.) The model puts no bound on the window. Amazon S3 offered eventual consistency for years, and on December 1, 2020, AWS announced that S3 “now delivers strong read-after-write consistency automatically for all applications.” (From the [AWS announcement](https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3-now-delivers-strong-read-after-write-consistency-automatically-for-all-applications/), posted December 1, 2020.) **Two machines, one file, four models** Machine B reads and caches `plan.md` at 0 s. Machine A writes v2, and its `write()` returns at 1 s, then closes the file at 2 s. If B reads again later: under linearizability, B sees v2 from 1 s. Under close-to-open, B sees v2 when it opens the file after 2 s. With a 30 s cache, B sees v1 until its entry expires at 30 s, or v2 from 2 s if it revalidates first. Under eventual consistency, B may see either version after 2 s, with no bound on when it settles. If B also writes its own edit, b, at 1.5 s and closes at 2.5 s, every one of the four models ends with b, and A’s edit is lost without an error. The models decide when B sees A’s write, not which write survives. Only a fifth arrangement, one writer at a time, keeps A’s v2: B’s write is refused until A gives its lease back. The second half of the figure matters most. Linearizable storage loses A’s edit as surely as eventual storage does, because the two machines never coordinated. A consistency model tells you when you’ll see another machine’s write. Stopping two machines from overwriting each other takes coordination, in which one of them is refused. ### Finding out a copy is stale A cache has two ways to learn that its copy is out of date: it can ask, or it can be told. AFS tried both, and its paper measured the difference. The AFS prototype asked. Its client checked a cached file’s timestamp with the server before every use, so “each open of a file thus resulted in at least one interaction with a server, even if the file were already in the cache and up to date.” Those checks, with status fetches, made up nearly 90 percent of the calls its servers handled. The revised AFS was told instead: “When a workstation caches a file or directory, the server promises to notify it before allowing a modification by any other workstation. This promise, called a Callback, dramatically reduces the number of cache validation requests received by servers.” For the redesign and its measurements in detail, see [Anatomy of a file create over a network](https://archil.com/guides/how-file-systems-work/anatomy-of-a-file-create). **Asking versus being told** A client opens `plan.md` five times. Asking on every open, as the AFS prototype did, sends a validity check and gets a reply each time: 10 messages. With a callback, the first open fetches the file along with the server’s promise, the next four opens send nothing, and when another client changes the file, the server sends one message to break the promise: 3 messages. Callbacks have a cost of their own, and the AFS paper names it. The server has to remember every promise it made, and “before modifying a file or directory, a server has to notify every workstation that has a callback on that file.” If a client’s record of its promises drifts from the server’s, the two can disagree without either noticing. After a reboot, a client treats everything in its cache as suspect. NFS chose to ask, on a timer. Its first published specification says the protocol “was intended to be as stateless as possible”, so that after a crash “a client need only retry a request until the server responds.” ([RFC 1094](https://www.rfc-editor.org/rfc/rfc1094.html), “NFS: Network File System Protocol Specification”, March 1989, section 1.3.) A server that keeps no promises can’t break any. The price is staleness, and the AFS authors noticed it at the time: they described NFS as a system “in which a file created on a workstation may not be visible on another workstation for 30 seconds.” ### Leases A callback has a hole. A server that promised to notify a client can’t notify one that crashed or that the network cut off. It can wait forever, or break its promise and let the client serve stale data. Gray and Cheriton’s answer was to put an expiry date on the promise: > A lease is a contract that gives its holder specified rights over property for a limited period of time. In the context of caching, a lease grants to its holder control over writes to the covered datum during the term of the lease, such that the server must obtain the approval of the leaseholder before the datum may be written. When a client writes, “the server must defer the request until each leaseholder has granted approval or the term of its lease has expired.” A lease holder that answers approves the write and drops its copy at once. One that doesn’t answer holds up the write only until its lease runs out. As long as the clocks behave, a crash or a lost message costs time, not correctness. **A lease running out** Client A caches `plan.md` at 0 s with a lease of 10 s, and client B asks to write the file at 4 s. If A answers, the server gets its approval and the write goes ahead at once. If A can’t be reached, the write waits until the lease expires at 10 s. If A’s clock also runs 20% slow, A believes its lease lasts until 12.5 s, so from 10 s to 12.5 s it can serve its old copy after B has written a new one. A server that allows for up to 30% drift waits until about 14.3 s instead, and A never serves a stale read. The lease term is a trade. Short terms limit how long a failed client can hold up writers, and long terms mean fewer renewals. Using file-access measurements from the V distributed system, the paper found that short terms are enough: across its workload, a 10 s term gave a server load within 5 percent of an infinite term. For how the paper placed AFS on the same scale, see [Anatomy of a file create over a network](https://archil.com/guides/how-file-systems-work/anatomy-of-a-file-create). Leases do need one thing that callbacks don’t: clocks that agree closely enough. The paper names the failure: “a server clock that advances too quickly can cause errors because it may allow a write before the term of a lease held by a previous client has expired at that client.” A client clock that runs slow does the same harm. Its fix is modest: “the correct functioning of leases requires only that clocks have a known bounded drift.” The server waits out that bound before it lets the write through, which is the allowance in the “A lease running out” figure. NFS version 4 put leases at its center. Its specification says lock state “is maintained at the server under a lease-based model”, and that a client which doesn’t renew in time can lose it: “all state associated with the client’s lease may be released by the server.” ([RFC 7530](https://www.rfc-editor.org/rfc/rfc7530.html), “Network File System (NFS) Version 4 Protocol”, March 2015, sections 1.4.5 and 1.5.) Version 4 also added delegations, in which the server can “delegate certain responsibilities to the client”, so that the client can “locally service operations” such as `OPEN`, `CLOSE`, `LOCK`, and `READ`. “Delegations can be recalled by the server,” which “requires that a callback path exist between the server and client.” A delegation that covers writes is ownership. The client holding it can accept writes into its own memory and send them later, because nobody else may write the file in the meantime. That’s what makes an owned file nearly as fast as a local one. The cost moves to the handoff: before another client can write, the owner has to send everything it holds and give the delegation back. ### Advisory locks Leases coordinate caches automatically. Locks are what a program asks for when it wants to coordinate with other programs, and on Linux they come in two kinds. `fcntl()` record locks cover byte ranges, and `flock()` covers a whole file. Both are advisory: “Advisory locks are not enforced and are useful only between cooperating processes.” (From the [fcntl\_locking(2) man page](https://man7.org/linux/man-pages/man2/fcntl_locking.2.html), which also covers lost locks and NFS.) Linux once had mandatory locking too. Its man page calls the implementation “unreliable”, and says the feature “is no longer supported at all in Linux 5.15 and above.” **An advisory lock on one machine** On one machine, process 1 takes a `flock` lock on `plan.md` and writes. Process 2 asks for the same lock and waits. `sed` edits `plan.md` without asking for any lock, and its write lands while process 1 holds the lock. Traditional `fcntl()` locks carry a trap even on one machine. “If a process closes any file descriptor referring to a file, then all of the process’s locks on that file are released.” A library that opens and closes your locked file for its own reasons drops your lock without telling you. Across machines, a lock is state that the server has to keep, and NFS was designed to keep none. The first specification left locking out: “Inherently stateful operations such as file or record locking, and remote execution, were implemented as separate services.” NFS version 3 described that separate service, the Network Lock Manager, as isolating “the inherently stateful aspects of file locking into a separate protocol.” ([RFC 1813](https://www.rfc-editor.org/rfc/rfc1813.html), “NFS Version 3 Protocol Specification”, June 1995, section 1.) Version 4 brought locks into the protocol itself, under the same leases as everything else. The Linux client maps `flock()` onto `fcntl()` locks over NFS: “Since Linux 2.6.12, NFS clients support `flock()` locks by emulating them as `fcntl(2)` byte-range locks on the entire file.” (From the notes in the [flock(2) man page](https://man7.org/linux/man-pages/man2/flock.2.html).) A lock held over a network can also be lost. The `fcntl_locking` man page describes a network partition “which lasts long enough for the server to assume that the client is no longer functioning.” Since Linux 3.12, an NFS version 4 client that loses its lock makes I/O fail for any process that “thinks” it still holds it, until that process closes and reopens the file. Linux once tried to recover such locks silently, and stopped by default “because of the attendant risk of data corruption.” ### Why locking across machines is hard The lost lock in the preceding section points at the real difficulty. A lock is a promise about the past: when the holder checked, it held the lock. By the time its write reaches the storage, that may no longer be true. Google’s Chubby paper describes the sequence precisely: “a process holding a lock L may issue a request R, but then fail. Another process may acquire L and perform some action before R arrives at its destination. If R later arrives, it may be acted on without the protection of L, and potentially on inconsistent data.” (Burrows, [“The Chubby lock service for loosely-coupled distributed systems”](https://research.google/pubs/the-chubby-lock-service-for-loosely-coupled-distributed-systems/), OSDI 2006, section 2.4.) The holder doesn’t have to crash for this to happen. Martin Kleppmann’s example is a garbage-collection pause: if it “lasts longer than the lease expiry period, and the client doesn’t realise that it has expired”, the client wakes and writes as if it still held the lock. (Kleppmann, [“How to do distributed locking”](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html), February 8, 2016.) Checking the lease immediately before writing doesn’t help, because the pause can fall between the check and the write. **A fencing token** Client 1 gets the lock with token 33, then pauses for garbage collection. Its lease expires, and the lock service gives client 2 token 34. Client 2 writes with 34, and the storage accepts it. Client 1 wakes and writes with 33, and the storage rejects the write, because it has already seen 34. Both sources reach the same fix. Each grant of the lock carries a number that only goes up, and every write carries the number of the grant it was made under. The storage remembers the highest number it has seen and rejects anything older. Chubby calls this a sequencer, “an opaque byte-string that describes the state of the lock immediately after acquisition”, including “the lock generation number”. Kleppmann calls it a fencing token, and notes that it “requires the storage server to take an active role in checking tokens.” For servers that can’t check, Chubby falls back on a lock-delay: when a holder disappears without releasing a lock, nobody else may take it for a period of up to one minute. For shared storage, this means a lock protects writes only if the component that accepts the write checks it. A lock the storage never sees protects only clients that cooperate and act on time. ### A worked example: Archil Archil makes a specific set of these choices, and the vocabulary of this page describes them. In conditional mode, any number of clients write the same paths at once. Their writes run on our servers, as they do over NFS, our servers apply them in the order they arrive, and the last write to reach the disk wins. A client that does most of the writing in a path can check it out instead, which our docs call a delegation. The holder answers `open`, `write`, and `rename` from memory, at local speed, and sends its changes to our servers about once a second and on every `fsync()`. Other clients can read the path in the meantime. Ownership is checked where the write lands: our servers refuse a write to a checked-out path from any other client, including through the disk’s S3 API, so a client that has lost its checkout can’t overwrite the new holder. Unlike a lease, a checkout doesn’t expire on its own. If the holder dies, someone takes the path over with `archil checkout --force`, and the writes the old client never sent are lost. (For more information, see [Sharing disks](https://docs.archil.com/concepts/sharing-disks) in our docs.) **A checkout changes hands** Client A has checked out `/repo`. A’s client answers its commands locally and sends its changes to the disk about once a second. Client B can read `/repo`, but its write fails as read-only. A runs `archil checkin`, which sends A’s changes and releases the path. Then B runs `archil checkout` and writes at local speed. Readers use a cache with a fixed lifetime. A client keeps entries for 30 s by default, and `archil invalidate-cache` drops them so the next read goes to our servers. For what this means for agents that share a workspace, see [Storage for agents](https://archil.com/guides/storage-for-agents). ## 5. A file system on an object store Source: https://archil.com/guides/how-file-systems-work/a-file-system-on-an-object-store · Part of How file systems work · Facts checked 2026-11-04 **In this chapter:** How a tree of files, with its modes, links, and renames, gets laid onto a flat map of immutable objects. By the end, you can read any product’s bucket layout and say what it gives up, what it costs to rename and list, and what happens when someone else writes to the bucket. A file system and an object store look alike from a distance. Both keep bytes under names, and both survive the machine that wrote them. Up close, they answer different questions. A file system is built for programs that change a few bytes in the middle of a file, rename it into place, and ask who may read it. An object store is built for whole objects, written once and read by anyone with the key. Every product that puts a file system in front of a bucket translates one into the other, and that translation decides what other tools find in the bucket. This page builds the translation one step at a time. It uses Linux as the file system and Amazon S3 as the object store, because Google Cloud Storage, Cloudflare R2, and many other stores accept the same API. ### Two interfaces A Linux file system is a tree. A directory holds names, and each name points at a file: its data, its mode bits, its owner, and its size. [From a path to an inode](https://archil.com/guides/how-file-systems-work/from-a-path-to-an-inode) builds this model from the disk up. Because names are separate from files, one file can have two names. The `link(2)` man page says the new name “may be used exactly as the old one for any operation; both names refer to the same file,” and that “it is impossible to tell which name was the ‘original’.” A program can write any byte range of a file at any time. And `rename(2)` is atomic: when it replaces a file, there is “no point at which another process attempting to access newpath will find it missing.” (From the Linux man pages for [link(2)](https://man7.org/linux/man-pages/man2/link.2.html) and [rename(2)](https://man7.org/linux/man-pages/man2/rename.2.html).) An S3 bucket is a map. Each key names one object, and an object is its data plus a little metadata. There are five calls that matter here: `PUT` stores a whole object, `GET` reads any byte range of one, `LIST` returns keys in order, `COPY` duplicates an object on the server, and `DELETE` removes one. S3’s documentation states the rules that make this simple to reason about. “Updates to a single key are atomic,” so a reader gets “either the old data or the new data, but never partial or corrupt data.” And “there is no way to make atomic updates across keys.” (From the [Amazon S3 data consistency model](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel).) Those two rules keep an object store simple to reason about. For a file system, they’re the whole difficulty. The following figure lines up what programs ask of a file system against what the object store offers. **Two interfaces** Nine file system calls beside the five calls an object store offers. Writing and closing a file maps onto `PUT`. Writing into the middle of a file also maps onto `PUT`, but only by rewriting the whole object. A read maps onto a ranged `GET`, and listing a directory maps onto `LIST` with a prefix. A rename maps onto two calls, `COPY` and then `DELETE`. Changing a mode or owner, making a hard or symbolic link, and making an empty directory have nothing to map onto. ### One object per file The most direct translation makes each file one object, with the file’s path as its key. The file `src/app.ts` becomes the object `src/app.ts`. Mountpoint for Amazon S3, s3fs, and Amazon S3 Files all work this way. Directories come for free, in a sense. S3 general purpose buckets have “a flat structure instead of a hierarchy,” and what looks like a folder is a shared prefix: if the keys `logs/date1.txt` and `logs/date2.txt` exist, there is a `logs` folder. (From AWS’s guide to [organizing objects by using folders](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html).) The file system rebuilds the tree from the keys. What this layout buys is the reason people choose it: every tool that speaks S3 reads the files where they sit. DuckDB, Spark, a training job, and `aws s3 cp` need nothing from the product that wrote them. What it costs is everything in the preceding figure that has no translation. The next sections go through those one at a time, and the following figure lets you try each one. Run a command, and watch both sides change and the requests each layout sends. **The tree and the bucket** A workspace on the mount (`app.db` at 64 MiB, `build.sh`, `notes.md`, and a `src` directory holding three files) and what the bucket holds for it, under two layouts. Each command sends the following to the bucket. With one object per file: `chmod +x build.sh` has nowhere to go unless the tool keeps modes as user-defined metadata, which S3 changes only by copying the object onto itself (one request). `ln -s notes.md README.md` becomes an 8-byte object, if the tool writes links as text. `ln notes.md notes-v1.md` becomes a second, independent 1 KiB object. `mkdir out` sends nothing, so the bucket has no `out/` until a file is written in it. `mv src code` copies and deletes each of the three objects: 6 requests. Writing 4 KiB into `app.db` uploads the whole 64 MiB object again. With blocks and a metadata database (JuiceFS’s layout): the first five commands change only rows in the database and send nothing to the bucket. Writing 4 KiB into `app.db` uploads one new 4 KiB block object and records in the database which range it covers. ### What an object can’t hold **Modes and owners.** An object has no mode bits and no owner in the POSIX sense, so a file system has to keep them somewhere else or do without them. Mountpoint does without: modes are set when you mount, and `chmod` and `chown` aren’t supported. S3 Files keeps owner, group, and permission bits as user-defined object metadata, and reads them back when it imports objects. (From Mountpoint’s [semantics document](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md) and AWS’s guide to [how S3 Files synchronization works](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-files-synchronization.html).) That works, with two costs. User-defined metadata is limited to 2 KB, and “after you upload the object, you can’t modify this user-defined metadata. The only way to modify this metadata is to make a copy of the object.” (From [Working with object metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html) in the S3 user guide.) So `chmod` on a 10 GiB file means a server-side copy of 10 GiB. **Symbolic links.** A symbolic link is a small file whose contents are a path, flagged so the kernel follows it. S3 has no such flag, so each product picks a representation. S3 Files supports links and stores each as an object. Its limits page caps a link’s target at 4,080 bytes, but AWS doesn’t document how the object encodes the link. (From the S3 Files page on [unsupported features, limits, and quotas](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-files-quotas.html).) Mountpoint doesn’t support links. Whichever choice a product makes, a plain S3 reader sees a short object and can’t tell it from a regular file without the product’s convention. **Hard links.** A hard link is two names for one file. A bucket has only keys, and two keys are two objects. Writing both keeps them equal until the next write through one name, which the other object never sees. S3 Files and Mountpoint don’t support hard links. **Empty directories.** A prefix exists only while some key carries it, so a directory with no files has no presence in the bucket. The S3 console works around this by creating “a 0-byte object” whose key is the folder name “plus a trailing forward slash.” Mountpoint takes the opposite approach: a new directory is local to that mount and isn’t preserved across a restart until a file is written into it. (From the S3 guide to [folders](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html) and Mountpoint’s [semantics document](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md).) A program that runs `mkdir -p build/out` and expects it to be there after a remount depends on which choice the product made. ### Rename is a copy and a delete S3 general purpose buckets can’t rename an object. S3 Express One Zone, a different bucket type, can rename a single file, but no bucket type can rename a directory. (From Mountpoint’s [semantics document](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md).) A file system with one object per file renames a file by copying the object to its new key and deleting the old one. `CopyObject` handles up to 5 GB in a single atomic action. A larger object needs a multipart copy, built from `UploadPartCopy` requests. (From the [CopyObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) API reference.) A directory rename repeats that for every object under the old prefix. AWS’s description of S3 Files puts it directly: “when you rename or move a directory, S3 Files must repeat this process for every object that shares that prefix.” Until it finishes, “the S3 bucket temporarily contains objects under both” prefixes, and objects written to the old prefix during the rename aren’t moved. (From AWS’s guide to [how S3 Files synchronization works](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-files-synchronization.html#s3-files-sync-rename-move).) On the mount, the rename returns at once. In the bucket, it’s a job. **A directory rename, object by object** A directory of five files is renamed from `runs` to `archive` on a mount that keeps one object per file. The command returns and the mount shows `archive` at once. In the bucket, each object is copied to its new key and then the old key is deleted, one file at a time: 10 requests for 5 files. Another reader that lists the bucket midway sees `archive/1.json` to `archive/3.json` alongside `runs/4.json` and `runs/5.json`. The cost grows with the number of files, not the number of renames. S3 Files warns when a file system covers a prefix so large that “a rename can take up to 4 hours (approximately up to 12 million objects).” That works out to about 12,000,000 ÷ 14,400 s, or roughly 830 objects per second. At that rate, renaming a directory of 100,000 files takes about 2 minutes. In requests, it’s 100,000 copies at S3 Standard’s $0.005 per 1,000 `PUT`, `COPY`, `POST`, or `LIST` requests, which is $0.50, and 100,000 deletes, which are free. (Request prices for S3 Standard in US East (N. Virginia), from the [AWS Price List](https://aws.amazon.com/s3/pricing/) published October 15, 2026. AWS’s pricing page states that `DELETE` requests are free.) The money is small. The window, during which a bucket reader sees a half-renamed tree, is what matters to anyone reading the bucket directly. ### Listing is a prefix scan To list a directory, a file system with one object per file asks S3 for the keys that start with that directory’s path. `ListObjectsV2` returns “up to 1,000” keys per request, in lexicographic order in a general purpose bucket. With a delimiter of `/`, keys that continue past the next slash are rolled up into one `CommonPrefixes` entry, so `ls src/` sees `src/lib/` as one subdirectory rather than every file inside it. If more results remain, the response says `IsTruncated` and carries a `NextContinuationToken` for the next request. (From the [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) API reference.) The following figure lets you send those requests yourself, with a page size small enough to show the chaining. **Listing as a prefix scan** Ten keys sit in a bucket in sorted order: `app.db`, `build.sh`, four keys under `node_modules/`, `notes.md`, and three keys under `src/`. `ls /` sends a `LIST` with a delimiter and gets back three files and two common prefixes, `node_modules/` and `src/`. `ls src/` gets back two files and one common prefix, `src/lib/`. `find src/` sends no delimiter and gets back all three keys under `src/`. `du node_modules/` gets back all four keys under it. With a page size of 3, any result longer than three entries takes more than one request, and each request carries the continuation token from the previous response. A delimiter keeps `ls` proportional to what one directory holds directly. A recursive walk has no such help. Installing the dependencies of one Next.js app with `npm ci` writes 18,602 files. (From Archil’s published [file system benchmark](https://docs.archil.com/reference/benchmarks/file-systems#archil-vs-juicefs-vs-amazon-s3-files). [Anatomy of a file create over a network](https://archil.com/guides/how-file-systems-work/anatomy-of-a-file-create) counts the round trips the same install pays.) Listing every key under that `node_modules/` takes at least 18,602 ÷ 1,000, rounded up, which is 19 requests, and they run one after another because each needs the token from the one before. Tools such as `du`, `find`, and `git status` walk trees like this. Products that put a file system in front of a bucket therefore keep their own copy of the listing, and that copy is where staleness comes from, as the next section explains. ### Consistency, before and after December 2020 For most of S3’s history, a write could be accepted before every reader could see it. On December 1, 2020, Jeff Barr described the old behavior as “a small time window where the data has been accepted and durably stored, but not yet visible to all GET or LIST requests.” That was hard on file systems, which assume that a file created a moment ago shows up in the next directory listing. Big data teams built workarounds: Amazon EMR built EMRFS Consistent View, and Hadoop developers built S3Guard. (From Jeff Barr’s post [Amazon S3 Update – Strong Read-After-Write Consistency](https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/) on the AWS News Blog.) S3Guard is worth a look, because it’s the same design this page reaches later. It kept a consistent database as “the store of metadata about objects in an S3 bucket,” to compensate for problems such as “newly created objects excluded from directory listings.” (From the Apache Hadoop page on [S3Guard](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/s3guard.html), which records its removal from the source in 2022.) The same day, AWS announced that “Amazon S3 now delivers strong read-after-write consistency automatically for all applications,” for list operations too, “at no additional cost.” (From AWS’s announcement of [strong read-after-write consistency](https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3-now-delivers-strong-read-after-write-consistency-automatically-for-all-applications/), posted December 1, 2020.) Since then, any `GET` or `LIST` that starts after a successful `PUT` returns sees that write. The Hadoop project removed S3Guard in 2022, because there was no longer anything for it to fix. **Read after write** Two lanes, before and since December 2020. In both, a writer’s `PUT` returns `200 OK` and the data is durably stored. Before December 2020, a `GET` right after an overwrite could still return the old version, and a `LIST` right after a new object was written could leave it out. Since then, a `GET` returns the new version and a `LIST` includes the new key. Strong consistency settles what one reader sees after one write. It doesn’t give a file system the rest of what it needs. S3 offers no locking for concurrent writers: “If two PUT requests are simultaneously made to the same key, the request with the latest timestamp wins.” Updates across keys still can’t be atomic, which is why a directory rename stays visible in pieces. (From the [Amazon S3 data consistency model](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel).) And the caches that products add on top bring a window back. Mountpoint, for example, can “return stale metadata for that object for up to 1 second, by default,” and longer when you turn on its metadata cache. (From Mountpoint’s [semantics document](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md).) ### Writing part of a file An object can’t be changed in place. To change any byte of it, you write a new object under the same key, and readers see the old one or the new one. For large objects, S3 offers multipart upload: you start an upload, send parts in any order and in parallel, and then complete it. Parts are 5 MiB to 5 GiB each, except the last, up to 10,000 of them. On completion, “Amazon S3 creates an object by concatenating the parts in ascending order,” and until then the object doesn’t exist under that key. A part can also be copied from a byte range of an existing object with `UploadPartCopy`, without sending those bytes again. (From AWS’s guides to [multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) and its [limits](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html).) Consider SQLite changing one 4 KiB page in the middle of a 1 GiB database. Rewriting the object uploads 1 GiB, which is 262,144 times the change, as [From a path to an inode](https://archil.com/guides/how-file-systems-work/from-a-path-to-an-inode) works out. A multipart copy does better: copy the range before the page and the range after it on the server, and upload one part of at least 5 MiB around the page. That’s five requests and 5 MiB uploaded, still 1,280 times the change. Each product that keeps one object per file handles this differently. Mountpoint uploads a new file in parts as it’s written, so writes “must always start from the beginning of the file and must be made sequentially,” and the object appears to other clients only after the file is closed or flushed. s3fs supports random writes, but “random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy.” (From Mountpoint’s [semantics document](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md) and the [s3fs README](https://github.com/s3fs-fuse/s3fs-fuse).) S3 Files absorbs writes on its own storage and exports a changed file after 60 seconds without writes, as one `PUT` of a new object version. **One SQLite update, three ways** SQLite changes one 4 KiB page in the middle of `app.db`. On JuiceFS, the page becomes a new block object under `chunks/`, and the map from file to blocks changes in the metadata database. On Mountpoint for Amazon S3, the write fails because writes must be sequential, and `app.db` is unchanged. On Amazon S3 Files, the change waits until the file has gone 60 seconds without writes, and then all of `app.db` is exported as a new object version. ### Blocks and a metadata database The other common design stops asking the object store to look like a file system. JuiceFS splits each file into 64 MiB chunks and each chunk into blocks of 4 MiB by default, and stores every block as an object. Its internals documentation gives the key format as `${fsname}/chunks/${hash}/${basename}`, where the basename is the slice ID, the block’s index, and its size. An example from the documentation is `jfstest/chunks/0/0/1_0_4194304`. Blocks are immutable: “all file edits is achieved through new blocks,” recorded as new slices that cover the old range. (From the JuiceFS [architecture page](https://juicefs.com/docs/community/architecture) and [internals documentation](https://juicefs.com/docs/community/internals/).) Names, directories, modes, links, and the map from each file to its blocks live in a metadata engine that you run, such as Redis or PostgreSQL. Everything that had no translation earlier now has a place. A mode is a field. A link is a row. An empty directory is a row. A rename changes one row in one transaction, however many files the directory holds, because no key names a path. A write in the middle of a file uploads only the new bytes. Switch the earlier figure, The tree and the bucket, to “Blocks and a metadata database” and run the same commands to see it. What this design gives up is the bucket. In JuiceFS’s words, after you write a file, “you cannot find the original file directly in the object storage.” Every reader goes through JuiceFS, either its client or its S3 gateway. The metadata engine becomes state as important as the data: the blocks mean nothing without the map. S3Guard used a separate database to correct listings. This design moves the whole namespace into one. ### Another writer in the bucket A bucket behind a mounted file system is still a bucket. Any program with credentials can write to it while the file system is mounted. Three questions tell you how a product copes: when the mount notices the change, what happens if both sides change the same file, and what happens to writes during a rename. The following table shows the answers for the two products on this page that document them. | | Mountpoint for Amazon S3 | Amazon S3 Files | | -------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | When the mount notices | Metadata can be stale for up to 1 second by default; directory listings are never stale | S3 Event Notifications update files whose data is on the file system; others update on next access | | Both sides change one file | No coordination between writers | The bucket wins; the file system’s version moves to a lost-and-found directory | | Writes during a rename | Directory renames aren’t supported | Objects written to the old prefix during a rename aren’t moved | In a chunked layout, the question mostly disappears: an object that another tool writes into the bucket isn’t a file to JuiceFS, because no row in its metadata engine points at it. ### A worked example: Archil Archil uses both designs, and you pick one per disk when you create it. A disk synced to your own bucket keeps one object per file, at its path, in S3, GCS, R2, or another S3-compatible store. A disk on Archil-managed storage, the default, uses Archil’s internal format in a bucket that Archil operates, and you read it through the disk’s own S3 API or a mount. (From our docs on [data ownership and portability](https://docs.archil.com/reference/data-ownership).) The synced disk is a hybrid of the two designs on this page. Its data is laid out like Mountpoint’s, so any S3 tool reads it. Its namespace lives in Archil, much as JuiceFS’s does in a metadata engine. That’s where mode bits, ownership, and extended attributes stay, and why a rename is atomic for any client that reads through Archil, even though the bucket shows partial results while the objects move. Links follow the choices described earlier: a symbolic link becomes a text file holding its target, and a hard link becomes a full copy. Writes reach the bucket in the background, combined so that several writes to one file become a single `PutObject`. They’re safe before then, because Archil replicates them across availability zones before `fsync()` returns. (From our docs on [data sources](https://docs.archil.com/concepts/data-sources) and [consistency](https://docs.archil.com/details/consistency).) In the other direction, Archil keeps a cache of the bucket’s contents on its servers to make reads fast, so an object written straight into the bucket appears on the disk after a few minutes. If a program writes a file through the disk while another writes the same object in the bucket, the result is undefined and can include lost or corrupted data. The disk’s own S3 API, which reads the disk directly, supports changes from both sides at once. **Written straight to the bucket** A pipeline outside Archil copies `batch.csv` into the bucket, and the mounted disk lists it a few minutes later, once its cached view of the bucket refreshes. Then an agent on the disk and a pipeline on the bucket both write `report.md`. That result is undefined, and data can be lost. ## 6. Copy-on-write, block by block Source: https://archil.com/guides/how-file-systems-work/copy-on-write · Part of How file systems work · Facts checked 2026-11-04 **In this chapter:** How a file system can copy a terabyte by writing one block, why the same design survives a power cut, and why deleting a snapshot takes more work than taking one. By the end you’ll be able to explain snapshots, clones, and their space accounting from the blocks up. A file system can copy a 10 TiB disk as fast as a 1 GiB disk, because the copy writes almost nothing. NetApp’s WAFL file system was doing it when its designers described it in 1994, and ZFS and btrfs are built on the same idea. To see how it works, start from what a file system is made of, and add one idea at a time. ### A file system is a tree of blocks A disk is an array of fixed-size blocks, each with a number, its address. A file system stores your files’ data in those blocks, and it stores the map of which blocks hold which file in blocks too. An inode records a file’s size, owner, and the addresses of its first blocks. For more information about inodes, see [From a path to an inode](https://archil.com/guides/how-file-systems-work/from-a-path-to-an-inode). A large file needs more addresses than fit in its inode, so the inode points at indirect blocks, which hold nothing but more addresses. Follow that chain upward and you reach one block that everything hangs from. The 1994 paper that describes WAFL puts it plainly: “A WAFL file system is best thought of as a tree of blocks.” (Dave Hitz, James Lau, and Michael Malcolm, [File System Design for an NFS File Server Appliance](https://www.usenix.org/legacy/publications/library/proceedings/sf94/full_papers/hitz.a), USENIX Winter 1994 Technical Conference, section 3.3.) Its root is the inode of the inode file, the file that holds every other inode. The leaves are the data blocks of every file. The file system has to find the root when it starts, so the root’s location is the one thing written at a fixed address. **A file system as a tree** A file system drawn as a tree of blocks. A root pointer at a fixed address names the root block. The root points to two metadata blocks (inodes), which point to four indirect blocks, which point to eight data blocks, `A` to `H`. Reading data block `F` means following four pointers down from the root: root, metadata, indirect, data. Every block in the file system is found the same way. Everything on this page follows from that shape. A file system whose blocks form a tree can describe its entire state with the address of one block. ### Updating a tree without overwriting it Appending one block to a file changes several blocks at once: the new data block, the indirect block that gains its address, the inode whose size grows, and the bitmap that records the block as used. Classic UNIX file systems such as the Berkeley Fast File System keep their metadata at fixed locations and update it in place. (Hitz, Lau, and Malcolm, section 3.2, compare WAFL with FFS, where “meta-data, such as inodes and bit maps, is kept in fixed locations.”) If the power fails after some of those blocks reach the disk and before the rest do, the disk holds a file system that never existed. Something has to prevent or repair the damage before the file system can be trusted again. A journal, which [Where fsync returns from](https://archil.com/guides/how-file-systems-work/where-fsync-returns-from) describes, is one answer. WAFL’s answer was different: never overwrite a block that the tree still points at. To change a data block, write the new version to an unused block. Its parent now has to point at the new address, so write a new copy of the parent too, with that one pointer changed. That parent’s parent changes for the same reason, and so on up to a new root. The final step is a single small write: move the root pointer from the old root to the new one. The technique came from databases. The WAFL paper notes that it is “well known for databases” and cites the shadow paging in IBM’s System R. (Hitz, Lau, and Malcolm, section 3.5: “See, for instance, \[Astrahan76] which describes the shadow paging technique used in System R.”) Ohad Rodeh, whose work on shadowed B-trees underlies btrfs, sums up the rule in two lines: the file system is “a tree of fixed-sized pages”, and you “never overwrite a page.” (Ohad Rodeh, [B-trees, Shadowing, and Clones](https://www.usenix.org/event/lsf07/tech/rodeh.pdf), slides from the 2007 Linux Storage and File Systems Workshop, slide 4. The btrfs design notes name this work as the basis of btrfs’s copy-on-write design: see [btrfs design](https://btrfs.readthedocs.io/en/latest/dev/dev-btrfs-design.html).) **One write, shadowed** One write under shadow paging, in a tree of fifteen blocks. First, the new data block, `C′`, is written to free space, where nothing points at it yet. Second, a new copy of each block above it is written, each with one pointer changed, up to a new root. Third, the root pointer moves to the new root in one small write. Fourth, nothing points at the old path any more, so its four blocks are freed. The other eleven blocks never moved, and the new tree shares them. The cost of a write is the depth of the tree, not its size. In a tree of fifteen blocks, a write writes four. In a real file system each block holds many pointers, so even a tree that covers a huge disk is only a few levels deep. A write still copies a handful of blocks, whether the disk holds a megabyte or a petabyte. Try it in the following figure: each block is labeled with its address, so you can see which blocks are new and which never moved. **Write to a shadowed tree** An interactive tree of fifteen blocks, each labeled with its disk address: root `#1`, metadata blocks `#2` and `#3`, indirect blocks `#4` to `#7`, and data blocks `A` to `H` at `#8` to `#15`. Selecting a data block writes it. Writing block `C`, for example, writes a new data block `C′` at `#16`, then new copies of its indirect block, its metadata block, and the root, at `#17` to `#19`. The root pointer moves to `#19`, and the old path, `#1`, `#2`, `#5`, and `#10`, is freed. Every write writes four blocks and frees four, and the other eleven never move. Copying a path for every write would still be a lot of writing if the file system did it one write at a time. It doesn’t. WAFL gathers “many hundreds of NFS requests” before it writes, so busy blocks, such as indirect blocks and blocks of the inode file, “are written only once per write episode instead of once per NFS request.” (Hitz, Lau, and Malcolm, section 3.4.) Rodeh’s slides list the same optimization: once a page is shadowed, it doesn’t need to be shadowed again until the next checkpoint. ZFS groups writes into transaction groups and, by default, writes one out at least every 5 seconds. (The `zfs_txg_timeout` parameter, documented in OpenZFS’s [zfs(4)](https://openzfs.github.io/openzfs-docs/man/master/4/zfs.4.html) manual page: “Flush dirty data to disk at least every this many seconds (maximum TXG duration).”) Many writes share one path copy and one root move. ### Crash consistency for free WAFL wrote anywhere to make snapshots possible, but the design brings something else with it. Until the root pointer moves, the tree it names is the old tree, untouched, because every new block went to free space. After the root pointer moves, it names the new tree, already complete on disk. No moment exists when the disk describes a half-applied change. There is one condition. The new blocks must reach stable storage before the root pointer that names them. ZFS writes this down in its source: before writing the uberblock, its name for the root pointer, it flushes the write cache of every disk it wrote, which “ensures that all blocks written in this txg will be committed to stable storage before any uberblock that references them.” (OpenZFS [`module/zfs/vdev_label.c`](https://github.com/openzfs/zfs/blob/1974111f45abdd9e9b76d111b3b046fa172e405b/module/zfs/vdev_label.c#L2289-L2292), in `vdev_config_sync()`. The same file explains that each device keeps “a redundant array of uberblocks”, and that ZFS picks the best one when it loads a pool.) btrfs does the same with its super block: the new tree blocks are flushed first, and “once the super block has been properly written to disk, the transaction is considered complete.” (From the [btrfs design](https://btrfs.readthedocs.io/en/latest/dev/dev-btrfs-design.html) notes, “Copy on write logging”.) **Power lost mid-commit** Two commits, each interrupted by a power cut. A commit writes the new data, writes the new parent blocks, flushes them to stable storage, and then moves the root pointer. In the first, power is lost after the parents are written and before the flush and the root move. On restart the root pointer still names the old root, and the disk holds the tree from before the write, whole: the new blocks are unreferenced free space. In the second, power is lost after the root pointer moved. On restart the disk holds the tree with the write, whole. The WAFL paper calls each committed tree a consistency point, and it treats one as a snapshot that has no name. WAFL took one at least every 10 seconds. After an unclean shutdown, it “simply reverts to the most recent consistency point,” which let a server with 20 GB of data reboot in about a minute without checking the file system. (Hitz, Lau, and Malcolm, section 3.5.) Reverting would lose the last few seconds of requests, so WAFL also logged each NFS request to battery-backed memory and replayed the log on restart. That split is worth remembering: the tree gives you consistency, and a log of recent operations gives you durability between commits. ### A snapshot is a root you keep After a commit, the old root and the blocks only it reached become free. Don’t free them, and you have a snapshot. The old root still names a complete tree, and it stays complete because the file system never overwrites a block that a tree reaches. That’s how WAFL takes one: “To create a virtual copy of this tree of blocks, WAFL simply duplicates the root inode.” A new snapshot “consumes no disk space except for the Snapshot inode itself.” (Hitz, Lau, and Malcolm, section 3.4.) btrfs describes its snapshots the same way: once dirty data is flushed, a snapshot “is instantaneous and only creates a new tree root copy in the metadata.” (From the [btrfs-subvolume(8)](https://btrfs.readthedocs.io/en/latest/btrfs-subvolume.html) manual page, “Performance”.) In ZFS, snapshots “can be created extremely quickly, and initially consume no additional space within the pool.” (From OpenZFS’s [zfsconcepts(7)](https://openzfs.github.io/openzfs-docs/man/master/7/zfsconcepts.7.html), “Snapshots”.) Notice the condition in the btrfs quote. A snapshot captures the tree, and the tree holds only what has been written to it. Writes still in memory have to be flushed first. btrfs warns that this step “may take some time.” WAFL’s 1994 snapshot procedure is mostly about that flush, and it finished “in less than a second” under normal loads. ZFS promises that a snapshot “includes all modifications to the dataset made by system calls that have successfully completed before that point in time.” (From OpenZFS’s [zfs-snapshot(8)](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-snapshot.8.html).) On a local file system, every completed write is in the kernel’s own memory, so the file system can flush it before the snapshot. Storage built from several machines has a harder version of this problem. A write can be acknowledged by one machine while the tree lives on others. What a snapshot includes then depends on which writes have reached the tree, and that’s a property to test, not to assume. ### Writable clones A snapshot is read-only. A clone is a second tree you can write to: a new root that points at the same blocks as a snapshot, and then moves on its own. In ZFS, “clones can only be created from a snapshot,” and “creating a clone is nearly instantaneous, and initially consumes no additional space.” In btrfs, snapshots are writable by default, and “they can be snapshotted again any number of times.” (From [zfsconcepts(7)](https://openzfs.github.io/openzfs-docs/man/master/7/zfsconcepts.7.html), “Clones”, and the [btrfs design](https://btrfs.readthedocs.io/en/latest/dev/dev-btrfs-design.html) notes, “Snapshots and subvolumes”.) Rodeh lists what a good cloning scheme needs: clones should share as many blocks as possible, a clone should take little time to make, you should be able to clone a tree many times, and you should be able to clone a clone. A full copy fails all four. With shadow paging, a write to either tree copies one path in that tree and leaves everything else shared. In the following figure, the checkpoint and `main` share a root. The branch has its own root, a copy of the original that points at the same children. Write to `main` or to the branch and watch which blocks stay shared. **Two trees that both move** An interactive figure with three trees of fifteen blocks: a read-only checkpoint, the writable `main`, and a writable branch. At the start, the checkpoint and `main` share root `#1`, and the branch has its own root, `#16`, which points at the same children, so every other block is reached by all three trees. Writing block `C` in the branch writes four new blocks, `#17` to `#20`, and frees only the branch’s old root, `#16`, because the checkpoint and `main` still reach the rest of the old path. Writing block `C` in `main` too writes four more and frees nothing. Each tree still sees fifteen blocks, and the disk holds 23. Deleting the checkpoint then frees the four blocks that only it reached: the old path to `C`. ZFS ties a clone to the snapshot it came from: “the original snapshot cannot be destroyed as long as a clone exists.” Its `promote` command reverses the relationship, so the clone becomes the parent and the old file system can be destroyed. That rule exists because of the problem in the next section. ### Deciding when a block is free In a file system with one tree, freeing is easy: when a write replaces a block, the old one is free as soon as the new root is in place. With snapshots and clones, the replaced block may still be reached by another tree. Before the file system can reuse it, it has to know whether any tree still points at it. Walking every tree to find out would take time in proportion to the whole disk. Three file systems answer the question three ways. #### One bit per snapshot WAFL keeps a block-map file with a 32-bit entry for every 4 KB block. “Bit 0 is set if the active file system references the block, bit 1 is set if the first Snapshot references the block, and so on. A block is in use if any of the bits in its block-map entry are set.” (Hitz, Lau, and Malcolm, section 4.1, and figure 5, which the following figure animates.) **A WAFL block-map entry over its life** One WAFL block-map entry over time, after figure 5 of the 1994 paper, with bits 3, 2, 1, and 0 shown left to right. At t1 the entry is 0000 and the block is unused. At t2 it is allocated for the active file system: 0001. Snapshot 1 is created at t3: 0011. Snapshot 2 is created at t4: 0111. At t5 the active file system deletes the block: 0110. Snapshot 3 is created at t6 and doesn’t include it: 0110. Snapshot 1 is deleted at t7: 0100. Snapshot 2 is deleted at t8: 0000, and the block is free again. The bitmap answers “is this block free?” with one lookup, but it moves the cost elsewhere. Taking a snapshot means copying the active bit into the new snapshot’s bit in every entry of the map. Deleting one means clearing that bit in every entry. Both are a pass over a map the size of the disk. The bits also cap the count: the paper describes keeping up to 20 snapshots online. Rodeh’s slides spell out the arithmetic: 32 bits per block allow 32 clones, and 256 clones would need 32 bytes for every block on the disk. #### Reference counts Rodeh’s design, which btrfs adopted, keeps a reference count for each block instead: the number of tree nodes that point at it. A count of zero means the block is free. (Rodeh, slides 24 to 30. The btrfs design notes: “Reference counting is the basis for the snapshotting subsystems.”) The trick is to update the counts lazily: - **To clone a tree**, copy its root and add one to the count of each of the root’s children. Nothing deeper changes yet, however large the tree. - **To write to a block whose count is above one**, shadow it as usual, subtract one from the old block’s count, and add one to the count of each of its children. The children now have one more parent. - **To delete a tree**, walk down from its root. At a block whose count is above one, subtract one and stop: another tree shares everything below it. At a block whose count is one, keep walking, and free the block on the way back up. The counts in the earlier figure, such as `×3`, are the same idea, counted in full rather than lazily. Deleting a tree visits the blocks that only it owns, plus the edge where sharing begins. That’s less than the whole disk, but it’s still real work, and it grows with how far the tree has diverged. btrfs does it in the background. Deleting a subvolume removes its name at once, but “the data blocks are removed later,” and this cleaning “can take some time depending on the amount of shared blocks.” (From the [btrfs-subvolume(8)](https://btrfs.readthedocs.io/en/latest/btrfs-subvolume.html) manual page, “Performance” and “delete”.) The design notes add that the walk records its progress, so a crash partway through doesn’t lose its place. #### Birth times ZFS needs no per-block counter. Each ZFS block pointer records, beside the block’s address and a 256-bit checksum of its contents, the transaction group in which the block was born. (The block pointer layout is in OpenZFS’s [`include/sys/spa.h`](https://github.com/openzfs/zfs/blob/1974111f45abdd9e9b76d111b3b046fa172e405b/include/sys/spa.h#L116-L177): “Each block is described by its DVAs, time of birth, checksum, etc.”) A transaction group number only grows, so it works as a clock, and a snapshot is a point on it. When the live file system stops using a block, one comparison decides its fate. If the block was born after the most recent snapshot, no snapshot can reach it, so it’s freed at once. Otherwise it goes on a dead list: the blocks the previous snapshot reaches that the file system after it doesn’t. (The rule is in `dsl_dataset_block_kill()` in OpenZFS’s [`module/zfs/dsl_dataset.c`](https://github.com/openzfs/zfs/blob/1974111f45abdd9e9b76d111b3b046fa172e405b/module/zfs/dsl_dataset.c#L304-L334).) Deleting a snapshot reads the dead list of the snapshot after it. Each block on that list that was born after the snapshot before it is now unreachable, and is freed. The rest are still held by that older snapshot. (In `dsl_destroy_snapshot_sync_impl()` in OpenZFS’s [`module/zfs/dsl_destroy.c`](https://github.com/openzfs/zfs/blob/1974111f45abdd9e9b76d111b3b046fa172e405b/module/zfs/dsl_destroy.c#L403-L431).) The work is in proportion to the dead list, which is what changed, not to the size of the disk. Try the rule in the following figure. **Birth times decide** An interactive timeline with snapshots at transaction groups 10, 25, and 40. A block’s life runs from its birth to the moment the live file system frees it, and a snapshot holds the block if it was taken during that life. For a block born at 14 and freed at 33, the snapshot at 25 holds it. When the live file system frees it at 33, it was born before the most recent snapshot, 25, so it goes on a dead list instead of being freed. Deleting the snapshot at 25 frees it: it’s on the dead list of the snapshot at 40, and it was born after the snapshot at 10, so no other snapshot holds it. Had it been born at 5, the snapshot at 10 would still hold it, and it would be kept. Set the three designs side by side and one pattern stands out. Taking a snapshot is cheap in all of them, because it only keeps a root. Deleting one is the expensive half, because the file system has to find exactly the blocks that no other tree reaches. WAFL pays with a pass over the whole map, btrfs with a walk of the deleted tree, and ZFS with a walk of a dead list. Any storage that offers forks has to solve deletion as well, and deletion is the harder half. ### Space accounting Once trees share blocks, “how much space does this snapshot use?” has two honest answers. ZFS reports both. A dataset’s `referenced` space is “the amount of data that is accessible by this dataset, which may or may not be shared with other datasets in the pool.” A snapshot’s `used` space is the space “referenced exclusively by this snapshot,” which deleting it would free. (From OpenZFS’s [zfsprops(7)](https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html), “Native Properties”.) Those two numbers can be far apart. Space that two snapshots share counts in neither one’s `used`, so the `used` values don’t add up to what the snapshots hold together. Delete one snapshot and its neighbors’ `used` can grow, because blocks they shared become theirs alone. For a clone, ZFS leaves out the space it shares with its origin. **What forks use** An interactive space meter for one 100 GiB checkpoint and several writable forks of it. Every fork references 100 GiB. The disk stores the checkpoint once plus what each fork rewrote: four forks that have each rewritten 10% store 140 GiB, against the 500 GiB the five trees show. Each fork uses 10 GiB, which deleting it would free. If the forks rewrote different blocks, every block of the checkpoint is still read by some fork, so the checkpoint uses nothing and deleting it frees nothing. If they all rewrote the same blocks, only the checkpoint still reads the old versions, and deleting it frees 10 GiB. A fork that has rewritten 100% of the disk costs as much as a full copy. Two lessons carry over to any storage that forks. First, a fork costs what it changes, so a fork that rewrites its whole workspace saves nothing over a copy. Second, the space you get back by deleting a fork is its own blocks, not its size. Deleting a snapshot that every clone still reads frees nothing at all. ### What copy-on-write costs The design isn’t free, and its costs follow from the same tree. Every write rewrites a path, which batching amortizes but doesn’t remove. Snapshots hold on to blocks that the live file system has replaced: the WAFL paper reports that keeping snapshots for a week consumed 10 to 20 percent of disk space in typical environments of the time. And deletion is deferred work, done in the background, so space comes back later than the command that freed it. ### A worked example: Archil’s branches Archil-managed disks, the default, use the same structure across a network. A disk’s state in object storage is a small root object that names immutable, content-addressed objects, which in turn name blocks of data. A flush writes new objects and then a new root, so it’s atomic for the same reason a WAFL consistency point is. A branch starts a new disk that reads its parent’s objects and cached chunks without copying them, and writes its own from then on. We bill each branch for the data it owns, so a shared chunk isn’t charged once per descendant. Disks synced to your own bucket keep ordinary objects at their paths instead, so this section doesn’t describe them. So a checkpoint or a branch takes the same time at any disk size. Our documentation gives median times of 13 ms for a checkpoint and 33 ms for a branch. (From [Branches and checkpoints](https://docs.archil.com/concepts/branches-and-checkpoints), [Limits](https://docs.archil.com/reference/limits), and [Metering](https://docs.archil.com/concepts/metering) in Archil’s documentation.) Deletion, the hard half, is what we haven’t shipped: today you can’t delete an individual branch or checkpoint. Branches also don’t merge back into their parent. A merge has to compare two trees, so its cost grows with the changes on each side. For how agents use forks to run several attempts from one starting state, see [Storage for agents](https://archil.com/guides/storage-for-agents). ## 7. How storage is metered Source: https://archil.com/guides/how-file-systems-work/how-storage-is-metered · Part of How file systems work · Facts checked 2026-11-04 **In this chapter:** Why a provider sells storage as reserved capacity or as bytes stored, what a GB-month measures, why cold data costs less to keep and more to read, and why, for agents, an idle machine costs more than the bytes it holds. A storage bill multiplies three things: a quantity of bytes, a length of time, and a rate. Then it adds charges for moving bytes around. Each of those parts answers a question the provider had to settle when it designed the service. What exactly is the customer reserving? How is it measured? What does it cost to keep a byte, and what does it cost to serve one? This page builds a storage bill up one part at a time, from the provider’s side of the counter. The prices are AWS’s list prices in us-east-1, read from AWS’s public price list, because AWS publishes the most of them. The mechanisms are the same at every provider. ### Reserved capacity and consumed capacity There are two ways to sell storage. The first is to rent out capacity. An Amazon EBS volume is a block device of a size you choose, placed in one Availability Zone, where, in AWS’s words, its data is “replicated across multiple servers”. A gp3 volume comes with 3,000 IOPS and 125 MB/s of throughput, and you can buy more of either. AWS bills the volume “by the amount of GB you provision per month until you release the storage”, whether you write a byte to it or not. (From [Amazon EBS features](https://aws.amazon.com/ebs/features/) and [Amazon EBS pricing](https://aws.amazon.com/ebs/pricing/). EBS bills in per-second increments, with a 60-second minimum.) From the provider’s side, a reserved volume is the simpler thing to plan for. The space and the performance are promised to you, so the provider sets them aside on specific servers and knows exactly what it has sold. You carry the risk of the part you don’t fill. A block device has no idea which of its blocks hold files, so it can’t bill you for less. The second way is to sell bytes stored. When Amazon S3 launched on March 14, 2006, it charged “$0.15 per gigabyte of storage per month”, and Amazon’s announcement put the model plainly: “S3 lets developers pay only for what they consume and there is no minimum fee.” Now the provider carries the empty space. It has to keep enough disks, servers, and network for every customer’s data to grow into, without knowing whose will. (From Amazon’s launch announcement, [Amazon Web Services Launches](https://press.aboutamazon.com/2006/3/amazon-web-services-launches).) A provider can afford that risk because it pools many customers, and their peaks rarely line up. If you reserve capacity, each customer sets aside its own peak, and the provider holds the sum of the peaks. If you pool, the provider needs room for the peak of the sum, which is smaller. **The sum of the peaks and the peak of the sum** Three tenants use storage that peaks at different hours of the day: 60, 50, and 70 GiB. Rented by size, each reserves its own peak, so 180 GiB are set aside. Pooled, the provider needs room only for the peak of their combined usage, about 91 GiB, because the peaks don’t line up. The numbers are an example. Pooling isn’t free. The provider still keeps headroom for growth and failures, and it runs a service that is always on for everyone. The per-byte rate pays for both, which is why a service that bills by bytes stored often has a higher rate than a volume. Amazon EFS Standard costs $0.30 per GB-month, and EBS gp3 costs $0.08. The two rates measure different things, though. One is per byte stored and the other is per byte reserved, so to compare them, divide the volume’s rate by how full you keep it. **What a stored gigabyte costs on a rented volume** A 100 GiB gp3 volume costs $8.00 a month however full it is. Divide that by the GiB actually stored to get the price per stored GiB: at 100% full it is the list price, $0.08, at 25% full it is $0.32, and at 10% full it is $0.80. EFS Standard costs $0.30 per stored GiB and S3 Standard $0.023, at any fill. The volume costs more per stored GiB than EFS Standard below about 27% full, and never gets as low as S3 Standard. ### What a GB-month measures “Per GB-month” sounds like a snapshot: look at the bucket once a month and multiply. It isn’t. Storage meters integrate. They measure how many bytes you stored for how many hours, then divide by the hours in the month. AWS’s own example, from the S3 FAQ, shows the whole calculation. A customer stores 4 GB for all of a 31-day month, and another 5 GB for the last 16 days: ```text 4,294,967,296 bytes × 31 days × 24 hours + 5,368,709,120 bytes × 16 days × 24 hours = 5,257,039,970,304 byte-hours 5,257,039,970,304 byte-hours ÷ 1,073,741,824 bytes per GB ÷ 744 hours = 6.581 GB-months ``` Three details in that example matter. A GB here is 2³⁰ bytes, 1,073,741,824, which most people would call a GiB. The month has 744 hours, because it has 31 days, so the same bytes cost slightly less per hour in a long month. And the 5 GB stored for half the month counts as half: 2.581 of the 6.581. (From the [Amazon S3 FAQ](https://aws.amazon.com/s3/faqs/). The S3 and EFS pricing pages both state that 1 GB is 2³⁰ bytes. The figures on this page use an average month of 730 hours, 365 × 24 ÷ 12.) The consequences follow directly. A burst of scratch data that exists for one hour costs one hour’s worth. Data you delete stops costing the moment it’s gone, unless the class has a minimum duration, which a later section covers. A provisioned volume runs the same integral over its size instead of its contents, so its meter never dips. **A byte-hours meter** The explorable shows one day of a workspace’s stored bytes, hour by hour, which you can redraw. It starts with a working day: 2 GiB overnight, rising to 8 GiB during the afternoon, then 3 GiB after cleanup. That day is 98 GiB-hours, an average of 4.1 GiB and a peak of 8 GiB. Billed by bytes stored, it is 98 ÷ 730 = 0.134 GiB-months, $0.0031 at S3 Standard’s rate. A 20 GiB gp3 volume bills 480 GiB-hours for the same day, 0.658 GiB-months, $0.053. A storage meter bills the average. A volume bills its size, which has to be at least the peak. ### Why cold data costs less to keep Most storage services sell several classes of the same bytes at different rates. The following table shows Amazon S3’s main classes in us-east-1. (Prices from AWS’s public price list for [Amazon S3](https://aws.amazon.com/s3/pricing/), read on 2026-11-04. Durations, sizes, and zones from [Understanding and managing Amazon S3 storage classes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html). Every class in the table is designed for 99.999999999% durability.) | Class | Storage, per GB-month | Retrieval, per GB | Minimum duration | Minimum billed size | Availability Zones | | ---------------------------- | --------------------- | --------------------------------------- | ---------------- | ---------------------------- | ------------------ | | S3 Standard | $0.023 | none | none | none | 3 or more | | S3 Standard-IA | $0.0125 | $0.01 | 30 days | 128 KB | 3 or more | | S3 One Zone-IA | $0.01 | $0.01 | 30 days | 128 KB | 1 | | S3 Glacier Instant Retrieval | $0.004 | $0.03 | 90 days | 128 KB | 3 or more | | S3 Glacier Deep Archive | $0.00099 | $0.02, after a restore that takes hours | 180 days | 40 KB of overhead per object | 3 or more | A provider’s cost for a byte has two parts: keeping it, and serving it. Keeping it means disks, plus the redundancy that protects them. Serving it means the disk time, servers, and network that answer reads. A class built for data that’s rarely read can spend less on both, and it charges a fee per read to cover the reads that do come. AWS doesn’t publish what hardware its cold classes run on. It does publish one comparison that isolates redundancy. S3 One Zone-IA and S3 Standard-IA have the same durability design and the same retrieval fee of $0.01 per GB. The difference AWS lists is that One Zone-IA keeps the data in one Availability Zone instead of at least three, so it doesn’t survive the loss of that zone. It costs 20% less. For why copies in separate zones survive failures that copies in one zone don’t, see [Where fsync returns from](https://archil.com/guides/how-file-systems-work/where-fsync-returns-from). Redundancy doesn’t have to mean full copies. Facebook’s 2014 paper on f4, its warm BLOB store, shows how a provider shrinks it for colder data. The authors found that photos and videos are requested far less as they age: for eight of the nine types they examined, week-old BLOBs were requested an order of magnitude less often than content less than a day old. Their hot store, Haystack, kept 3.6 bytes on disk for each byte of data. f4 stored warm BLOBs with Reed-Solomon (10, 4) erasure coding instead, and cut that “effective-replication-factor” to 2.8, and then to 2.1 with XOR coding across datacenters. (From Muralidhar and others, [f4: Facebook’s Warm BLOB Storage System](https://www.usenix.org/conference/osdi14/technical-sessions/presentation/muralidhar), OSDI 2014. At the time, f4 stored over 65 PB of logical BLOBs.) **Three copies against a coded stripe** Ten blocks of data kept as three full copies take 30 blocks of disk, 3.0 times the data. With Reed-Solomon (10, 4) coding they take 14 blocks, 1.4 times the data: ten data blocks and four parity blocks, any ten of which rebuild the rest. When a block is lost, three copies rebuild it by reading one other copy, and the coded stripe reads ten blocks. The saving has a price. A coded stripe needs ten reads to rebuild one lost block where a copy needs one, and every read of cold data competes for disks that were bought for capacity, not speed. That’s the trade a cold class offers: cheaper bytes, in exchange for more expensive reads. #### The break-even between keeping and fetching The arithmetic for choosing between two classes is old. In 1985, Jim Gray and Franco Putzolu at Tandem Computers asked when it pays to keep a record in memory instead of reading it from disk each time. A disk that served 15 accesses per second cost about $1,000 per access per second, plus about $1,000 more in CPU and channel. A kilobyte of memory cost $5. So a 1 KB record that saved one disk access per second was worth keeping in memory, and the break-even came at one access every 400 seconds. They called it the five-minute rule: “Pages referenced every five minutes should be memory resident.” (From Gray and Putzolu, [The 5 Minute Rule for Trading Memory for Disc Accesses and the 5 Byte Rule for Trading Memory for CPU Time](https://dsf.berkeley.edu/cs286/papers/fiveminute-tr1986.pdf), Tandem Technical Report 86.1, May 1985, revised February 1986.) The same reasoning picks a storage class. Moving a GB from S3 Standard to Standard-IA saves $0.0105 a month. Reading it back once costs $0.01. So if you read the whole object about once a month or more, Standard is cheaper. AWS’s own table says the same: Standard is for data accessed “more than once a month”, and Standard-IA for data accessed “once a month”. Moving on to Glacier Instant Retrieval saves another $0.0085 a month and costs $0.02 more per read, so it pays if you read the object less than about once every 2.4 months. AWS describes that class as “accessed once a quarter”. ### When data moves between classes A provider can move data to a colder class by its age or by its last access. The choice matters, because the two rules disagree about the same file. - **By age.** S3 Lifecycle rules count “the number of days since object creation”. Reading an object doesn’t reset that count. f4’s finding, that age predicts temperature, is why an age rule suits data like photos. - **By access.** EFS lifecycle management moves a file to Infrequent Access after 30 days, by default, without access, and “whenever a file in Standard is accessed, the lifecycle management timer is reset”. S3 Intelligent-Tiering moves an object after 30 consecutive days without access, and again after 90. It charges a monitoring fee per object instead of a retrieval fee. (From [Lifecycle configuration elements](https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html), [EFS lifecycle management](https://docs.aws.amazon.com/efs/latest/ug/lifecycle-management-efs.html), and [Amazon S3 storage classes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html). EFS keeps every file’s metadata in Standard.) An agent’s workspace breaks the age rule. The repository and dependencies it created on the first day are the files it reads in every session, so an age rule would move the hottest files first. An access rule fits better, but a workspace the agent opens every day never goes 30 days without access, so it never moves at all. Every move also costs something. S3 charges a request per object for each transition, and an object that leaves a class before its minimum duration is billed for the rest of it. A lifecycle rule can’t even be written to move objects out early. In AWS’s example, a rule that moves objects to Glacier Instant Retrieval after 4 days can’t move them to Deep Archive until day 94. (From [Transitioning objects using Amazon S3 Lifecycle](https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-transition-general-considerations.html). Transitions cost $0.01 per 1,000 objects into Standard-IA, $0.02 into Glacier Instant Retrieval, and $0.05 into Deep Archive.) **A year of a lifecycle rule** A million objects follow a common lifecycle rule for a year: Standard-IA at day 30, Glacier Instant Retrieval at day 90, and Deep Archive at day 180. At 1 MiB each, about 977 GiB in all, never read and kept all year, the rule costs about $145 against $270 left in Standard, and $80 of that is the three transition requests per object. Read once a month, the retrieval fees and restore requests bring it to about $1,010 against $274. Deleted on day 100, the objects still bill the remaining 80 days of Glacier Instant Retrieval’s 90-day minimum, about $10, and the rule costs $88 against $74. At 4 KiB each, S3 Lifecycle doesn’t move the objects by default, and they cost about $1 for the year in Standard. Forcing the move costs $80 in transitions and bills each object as 128 KB, about $86 in all. ### Minimum sizes and durations The minimums in the table come from costs that don’t scale with an object’s size. Every object needs an index entry and metadata, and every transition is a request. For a large object, those costs vanish into the per-byte rate. For a small one, they don’t, so the provider sets a floor. S3’s floor is the minimum billed size in the table. The archive classes, Glacier Flexible Retrieval and Deep Archive, add metadata to each object instead: 32 KB at the archive rate and 8 KB at Standard’s. EFS’s price list has a line for “Infrequent Access usage from small file rounding”, which applies to files under 128 KB. Since September 2024, S3 Lifecycle doesn’t move objects under that size by default. AWS explains that the transition request “can outweigh the storage savings”. The minimum duration does the same for time. A cold class is priced on the assumption that data stays, so leaving before the table’s minimum duration bills the rest of the stay. Google Cloud Storage uses the same structure, with minimums of 30 days for Nearline, 90 for Coldline, and 365 for Archive. (From [Google Cloud Storage classes](https://docs.cloud.google.com/storage/docs/storage-classes).) **Billed for more than you store** In S3 Standard-IA, each object is billed as at least 128 KB. Ten thousand 4 KiB files store 39 MiB and bill as 1.22 GiB, 32 times their size. Each object is also billed for at least 30 days: one deleted on day 10 still bills the remaining 20. A software workspace holds many small files. `npm ci` for one Next.js app writes 18,602 of them, so in a cold class they’d pay 18,602 minimums. (The file count is from Archil’s published [file system benchmark](https://docs.archil.com/reference/benchmarks/file-systems#archil-vs-juicefs-vs-amazon-s3-files).) For what those files cost in round trips rather than dollars, see [Anatomy of a file create over a network](https://archil.com/guides/how-file-systems-work/anatomy-of-a-file-create). ### Requests, throughput, and egress Bytes at rest are only part of a provider’s cost. Each request needs authentication, an index lookup, and disk time, whatever its size, so object stores charge per request. S3 Standard charges $0.005 per 1,000 PUT, COPY, POST, or LIST requests and $0.0004 per 1,000 GETs, so a write costs 12.5 times as much as a read. [A file system on an object store](https://archil.com/guides/how-file-systems-work/a-file-system-on-an-object-store) shows how many requests a rename or a listing sends. For small files, requests can cost more than storage. Uploading that dependency tree as objects takes 18,602 PUTs, about $0.09. Doing it once a day for 30 days costs about $2.79, more than a month of storing 100 GiB in S3 Standard, which costs $2.30. File systems meter the same work as throughput. With Elastic Throughput, EFS charges $0.03 per GB read and $0.06 per GB written. A gp3 volume folds its throughput into the reservation: a baseline of IOPS and throughput comes with the size, and more is reserved and billed separately, like the size itself. Moving data between places has its own meter. Data into AWS is free. Data out to the internet costs $0.09 per GB for the first 10 TB a month, and data between Availability Zones in a region costs $0.01 per GB. Copying a 100 GiB workspace out once costs about $9, which is about four months of storing it in S3 Standard. (From AWS’s public price list for [AWS data transfer in us-east-1](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSDataTransfer/current/us-east-1/index.json) and for [Amazon EFS](https://aws.amazon.com/efs/pricing/), read on 2026-11-04.) ### Compute that waits Compute is sold with the same two models as storage, and agents mostly buy it the expensive way. A sandbox reserves its CPU and memory. E2B bills those per second for as long as the sandbox runs, used or not. Vercel Sandbox bills memory for as long as it’s provisioned, idle time included, and bills CPU only while it’s actively used. (From [E2B pricing](https://e2b.dev/pricing), $0.000014 per vCPU-second and $0.0000045 per GiB-second, and [Vercel Sandbox pricing](https://vercel.com/docs/sandbox/pricing), both read on 2026-11-04.) Put an idle sandbox hour next to a storage month. One hour of a 2 vCPU, 4 GiB sandbox at E2B’s list price costs about $0.17. That buys 7.2 GiB-months of S3 Standard. **One idle hour, priced in storage** One idle hour of a 2 vCPU, 4 GiB sandbox at E2B’s list price, about $0.17, buys 7.2 GiB-months of S3 Standard at $0.023 each, or 2.1 GiB-months of EBS gp3. A 5 GiB workspace kept in S3 Standard for a month costs $0.115, less than that one hour. Agents leave machines waiting for a long time. Archil measured 384 coding-agent loops from its own engineering team, and found that a sandbox that stopped after 15 idle minutes still sat idle for 65% of the time it ran. (The method is in [Do AI agents need a sandbox running for the whole session?](https://archil.com/article/do-ai-agents-need-a-sandbox-running-for-the-whole-session)) At those rates, the machine kept up to hold a workspace costs far more than the bytes in it. ### A worked example: Archil’s meter Archil bills a disk by consumption, with an access rule of one hour. Data counts as active while it sits in our high-speed layer: data you read stays active for up to 1 hour after your last access, and data you write for up to 1 hour after we flush it to the disk’s data source. We measure active data every minute and bill the time-weighted average, so 100 GiB that’s active for half a month is 50 GiB-months, at $0.20 per GiB-month. Reading metadata with `ls`, `stat`, or `find` doesn’t make file data active. Each metadata write, such as a rename or `chmod`, counts as 32 KiB until it reaches the data source. A client that has checked out a directory coalesces those writes first, so a `git clone` of a 15 MB repository meters as about fifteen 1 MiB writes rather than thousands of metadata operations. (From [Metering](https://docs.archil.com/concepts/metering) and [Pricing & Billing](https://docs.archil.com/administration/billing) in our docs.) Under the active meter sits the whole disk. On a disk we manage, it bills archive storage at $0.025 per GB-month. On a disk synced to your own bucket, there’s no archive charge, and your cloud provider bills the bucket at its own rate. Our price list has no retrieval fee and no minimum object size or duration. Data leaving Archil costs $0.07 per GB, and transfer to our own tool calls and sandboxes is free. For how this changes what agents should pay for, see [Storage for agents](https://archil.com/guides/storage-for-agents).