# 3. Anatomy of a file create over a network

Source: https://archil.com/guides/how-file-systems-work/anatomy-of-a-file-create · Part of How file systems work · Facts checked 2026-11-04

**In this chapter:** What a program asks for when it creates one small file, why a network file system has to ask a server before it can answer, and how callbacks, leases, and delegations let a client answer from its own memory once it holds the right to.

Creating a file looks like one action, and on a laptop it costs about as much as one. Over a network it can cost several round trips, and a job that creates thousands of files pays that cost thousands of times. This page builds up why, one step at a time. It starts with a create on a single machine, sends the same create to a server, and then follows the three ideas that let a client stop asking, which are callbacks, leases, and delegations.

## One file on one machine

A program that unpacks a package creates each file with a handful of system calls:

```c
fd = open("node_modules/react/index.js", O_CREAT | O_WRONLY, 0644);
write(fd, buf, 4096);
fchmod(fd, 0644);
close(fd);
```

Each call changes a different part of the file system. `open` with `O_CREAT` adds a name to a directory and allocates an _inode_, the record that holds a file’s owner, mode, size, and timestamps. For more information about inodes and directories, see [From a path to an inode](https://archil.com/guides/how-file-systems-work/from-a-path-to-an-inode). `write` puts 4 KiB of data into the file. `fchmod` changes the mode in the inode. `close` releases the descriptor.

On Linux, none of these calls waits for the disk. The kernel keeps directory entries, inodes, and file data in memory: the dentry cache, the inode cache, and the page cache. Each call changes memory and returns. The disk catches up later. ext4, for example, commits its journal every 5 seconds by default, and data reaches the disk when the kernel writes back dirty pages, or when a program calls `fsync()`. [Where fsync returns from](https://archil.com/guides/how-file-systems-work/where-fsync-returns-from) follows that path in detail.

**A create on a local disk**

A program creates a file with four calls: open with O\_CREAT, write 4 KiB, fchmod, and close. Each call changes something in the kernel’s memory, a directory entry, a cached page, or an inode’s mode, and returns at once. The disk is written later, when ext4 commits its journal, every 5 seconds by default.

This works because one kernel owns this file system, so it’s the only place that can change a name in that directory. It never has to ask anyone whether `index.js` already exists.

## The same create, sent to a server

Sun’s Network File System puts a file system behind a server, to give many machines “transparent remote access to shared files across networks.” The specification of its second version, published as [RFC 1094](https://www.rfc-editor.org/rfc/rfc1094) in March 1989, states a goal that shaped everything after it: “The NFS protocol was intended to be as stateless as possible.” A stateless server keeps no record of its clients. If it crashes and restarts, clients retry their requests and carry on, with no state to rebuild.

The NFS client in the kernel turns each system call into a remote procedure call. NFS version 3, specified in [RFC 1813](https://www.rfc-editor.org/rfc/rfc1813), has a procedure for each kind of change, and the small file in the preceding section needs four of them:

- `CREATE` makes the name and the inode.
- `SETATTR` sets the mode that `fchmod` asked for.
- `WRITE` sends the 4 KiB of data. NFSv3 lets the server hold this write in memory, marked `UNSTABLE`.
- `COMMIT` asks the server to put that data on stable storage. Clients send the pending write and the commit when the file is closed. RFC 7530 makes that a rule for NFSv4 and notes that many NFSv3 clients already follow it.

`CREATE` and `COMMIT` are on RFC 1813’s list of synchronous procedures: the server writes the change to stable storage before it replies. So each round trip includes a durable write on the server, as well as the network.

The cost of a file is now the number of calls that wait, multiplied by the distance to the server. The following explorable does that multiplication. Move the server closer or farther away, and choose a job.

**Round trips per file**

Creating one small file takes four calls. Over NFS, three of them wait for the server: CREATE when the file is opened, SETATTR when its mode is set, and WRITE and COMMIT when it’s closed, for four round trips in all. A client that holds the directory answers all four itself. At 0.5 ms per round trip, an `npm ci` of 18,602 files waits for 18,602 × 4 × 0.5 ms, about 37 seconds. At 40 ms, it waits about 50 minutes. Real clients overlap some calls, so these totals show the size of the problem, not a prediction.

Real clients overlap some of this work. A package manager runs several unpacks at once, and the kernel can have several requests in flight. But each file’s own calls happen in order, and a job of thousands of files still pays thousands of waits.

## Why only the server can decide

Suppose two machines, A and B, both run `touch hello.txt` in the same shared directory at the same moment. Only one of them can create the file. A’s kernel doesn’t know about B’s request, and B’s kernel doesn’t know about A’s. The server is the only place where both requests arrive, so it’s the only place that can decide.

NFS gives the server the tools to decide. A `CREATE` request can be `GUARDED`, which tells the server to check for a file with the same name first and fail with `NFS3ERR_EXIST` if one is there. It can also be `EXCLUSIVE`, which handles a retried request correctly by storing a verifier with the new file. RFC 1813 adds that after an exclusive create, the client “must issue a SETATTR to set the correct file attributes,” which is one reason `SETATTR` follows `CREATE` so often.

**Who gets hello.txt?**

Machines A and B both create hello.txt. Only the NFS server can decide who wins, so both ask. A wins, and B is told the file exists. Then A creates four more files that nobody else wants, and each one still makes a round trip, because the server has no way to know that nobody else is looking.

The second half of that story is the expensive half. A goes on to create files that no other machine will ever touch, and every create still crosses the network. A stateless server keeps no record of who is working in which directory, so it can’t tell A that it has the directory to itself. A can’t know that either. Asking is the only safe choice, so A asks every time.

## What a client may cache

A client can’t decide a create alone, but it can remember what it has read. NFS clients cache file attributes and directory contents, and treat them as current for a while. Linux’s defaults, from the `nfs(5)` man page, are 3 to 60 seconds for a file’s attributes and 30 to 60 seconds for a directory’s.

Caching raises the question of when one client sees another’s change. NFS clients answer it with _close-to-open consistency_. When a program closes a file, the client sends its pending writes to the server. When a program opens a file, the client asks the server for the file’s current attributes, “regardless of the freshness of the file’s cached attributes,” in the man page’s words. If A closes a file before B opens it, B sees A’s changes.

**Close-to-open, and a cached listing**

Client A writes notes.txt and closes it, which sends WRITE and COMMIT to the server. Client B opens the file. The open always sends GETATTR, so B reads A’s new version. Then A creates new\.txt. B lists the directory, and its client answers from cached directory attributes, so new\.txt is missing. Linux trusts those attributes for at least 30 seconds by default.

This is a good fit for files that one machine writes and another reads later, which is most sharing. It doesn’t make anything faster for a create: the close still waits for the server, and the next open still asks. It also leaves a window in which a listing is stale. The Andrew File System paper described NFS as a system “in which a file created on a workstation may not be visible on another workstation for 30 seconds.”

## Callbacks: the Andrew File System

The Andrew File System was built at Carnegie Mellon, which planned for it to span more than 5,000 workstations. John Howard and six colleagues described its first two versions in “[Scale and Performance in a Distributed File System](https://www.cs.cmu.edu/~coda/absdir/s11-abstract.html),” published in _ACM Transactions on Computer Systems_ in February 1988. The paper is a careful record of a design that didn’t scale, the measurements that showed why, and the change that fixed it.

AFS caches whole files on each workstation’s local disk. A process called Venus manages the cache and talks to the servers, called Vice. In the prototype, Venus treated every cached file as suspect: “Before using a cached file, Venus would verify its timestamp with that on the server responsible for the file. Each open of a file thus resulted in at least one interaction with a server, even if the file were already in the cache and up to date.” Programs that called `stat` before opening a file made it worse, because each `stat` checked the cache too.

Two calls made up nearly 90% of the traffic to the servers: `TestAuth`, which validated a cache entry, and `GetFileStat`, which fetched the status of a file absent from the cache. Only 6% of calls moved file data. Performance was usually acceptable up to about 20 active users per server. In the team’s benchmark, `TestAuth` response times rose rapidly once more than five workstations ran it at the same time.

The revised AFS turned the question around. In the paper’s words, “When a workstation caches a file or directory, the server promises to notify it before allowing a modification by any other workstation. This promise, called a Callback, dramatically reduces the number of cache validation requests received by servers.” Venus now assumes a cached entry is valid unless told otherwise, so an open of a cached file needs no message at all.

**Checking every time, or being told**

In the AFS prototype, Venus checks with the server on every open of a cached file, so six opens make six TestAuth calls that get the same answer each time. In the revised AFS, the first open fetches the file and the server records a callback, a promise to notify Venus before any other workstation changes it. The next four opens stay local. When another workstation stores the file, the server breaks the callback, Venus drops its copy, and the sixth open fetches the file again: two calls in all.

The numbers moved a long way. The prototype ran the benchmark 70% slower than a workstation with a local disk, and the revised system ran it 19% slower. In the prototype, the benchmark took more than four times as long at a load of 10 as at a load of 1. In the revised system it took 36% longer at that load, and less than twice as long at a load of 20. The authors concluded that their goal of 50 users per server had been met. Callbacks also let workstations resolve path names from their caches, because a cached directory with a callback on it can be trusted.

Callbacks cost something too. The server now keeps state for every cached file, and must notify every holder before it allows a change. A rebooted workstation can’t trust anything in its cache and has to revalidate each entry the first time it’s used. And callbacks covered reading, not writing names: “For reasons of integrity, modifications to a directory are made directly on the server responsible for that directory.” In AFS, a create still went to the server.

## Leases: a promise with an expiry

A callback is a promise with no end date, which is a problem when machines fail. If the server can’t reach a workstation to break a callback, it can wait forever, or go ahead and let the workstation read a stale copy. In 1989, Cary Gray and David Cheriton at Stanford proposed a way out in “[Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency](https://dl.acm.org/doi/10.1145/74850.74870),” published in the proceedings of the 12th ACM Symposium on Operating Systems Principles.

Their definition is short: “A lease is a contract that gives its holder specified rights over property for a limited period of time.” A cache that holds a lease on a file can serve reads of it without asking the server. When another client wants to write, the server must get the leaseholder’s approval, or wait until the lease has expired. A lease can cover the file’s name and permissions, not only its contents, so a rename counts as a write.

The expiry is what makes leases tolerate failure. If a leaseholder crashes or the network splits, the server waits out the lease and then lets the write proceed, knowing nobody can still be relying on the old copy. A failure costs the writer time, and no client reads a stale copy. A server that restarts only has to remember the longest lease it granted, and hold writes for that long.

**Approval, or the end of the term**

Client A reads a file and gets a lease with it, so its next reads need no message. Client B writes the file. The server asks A to approve, A drops its copy and approves, and the write is applied. Then A takes a new 10-second lease and crashes. B writes again. The server gets no answer from A, so it waits until A’s lease term is over and then applies the write. The write is late, but no client ever reads a stale copy.

The paper then asks how long a lease should last, and the answer is shorter than you might expect. In their model, built on file access data from Stanford’s V system, a 10-second term on a file cached by one client cut the traffic spent on consistency to 10% of what a zero-length term needed. Longer terms added little. Gray and Cheriton placed AFS on the same scale: its prototype amounted to a lease term of zero, and its revised version to a term of infinity. Short terms also limit _false sharing_, where a writer has to wait for a lease that’s held by a client no longer using the file.

## Delegations in NFSv4

NFS version 4 gave up statelessness. Its specification, now [RFC 7530](https://www.rfc-editor.org/rfc/rfc7530), keeps a client’s opens, locks, and other state on the server under a single lease that the client renews. On that foundation it adds _delegations_, a mechanism “that allows many decisions normally made by the server to be made locally by clients.”

An open delegation hands a client the right to handle opens and closes of one file by itself. With a read delegation, other clients may not change the file. With a write delegation, the client may change the file’s data, and only the attributes tied to it: size, modification time, and the change attribute. While it holds either, the client “does not send OPENs or CLOSEs to the server.” When another client wants the file, the server recalls the delegation with a callback, as AFS did. The holder sends its modified data back and returns the delegation, and the second client’s request waits until it does.

NFSv4.1, now [RFC 8881](https://www.rfc-editor.org/rfc/rfc8881), added _directory delegations_ for exactly the case on this page. The RFC names compilation as a workload with many lookups of files that don’t exist yet, and credits AFS and DFS with showing that state about directory contents “can greatly reduce network traffic.” Directory delegations are optional for servers, and they stop short of writing: “The delegation is read-only and the client may not make changes to the directory other than by performing NFSv4.1 operations that modify the directory or the associated file attributes so that the server has knowledge of these changes.” A client with a directory delegation can list the directory and miss on lookups from memory. It still sends every create to the server.

The following table summarizes what each mechanism lets a client do without asking the server.

| Mechanism                    | Reuse cached data and names       | Open and close a file             | Write a file’s data         | Create a name in a directory |
| ---------------------------- | --------------------------------- | --------------------------------- | --------------------------- | ---------------------------- |
| NFSv3 attribute cache        | For a few seconds, then ask again | No                                | No, flushed on close        | No                           |
| AFS callback                 | Until the server breaks it        | Opens, yes                        | Yes, stored on close        | No                           |
| Lease                        | Until the term ends               | Yes, if the lease covers the name | No, writes go to the server | No, writes go to the server  |
| NFSv4 write delegation       | Yes                               | Yes                               | Yes                         | No                           |
| NFSv4.1 directory delegation | Yes, for the directory            | No                                | No                          | No                           |
| Holding the directory        | Yes                               | Yes                               | Yes                         | Yes                          |

## Answering a create from memory

The last row is what a small-file job needs. For a client to answer `open(O_CREAT)` by itself, the server has to hand it the directory: the exclusive right to add and remove names there. Everything else follows the patterns this page has already built.

- The right has to be exclusive, like a write delegation, because two clients deciding names in one directory is the conflict the server exists to prevent.
- The server needs a way to take it back. When another client asks for the directory, the server recalls it, the holder sends every change it made, and only then does the server answer the second client.
- The changes still travel, but not one at a time. A client that created 500 files sends them to the server together, so 500 waits become one.
- Durability is a separate question. A create that exists only in the holder’s memory is not yet safe, exactly as a write on a local disk isn’t safe before `fsync()`. For what has to happen before `fsync()` returns, see [Where fsync returns from](https://archil.com/guides/how-file-systems-work/where-fsync-returns-from).

Try it in the following explorable. Create a few files from A while nobody holds the directory, then hand A the directory and create a few more. Then create a file from B.

**Who decides a create?**

Two clients create files in one shared directory. When nobody holds the directory, every create from either client is one round trip, and the server decides each one. A second create of hello.txt fails because the name exists. When client A holds the directory, A decides its own creates in memory with no round trips, and its new names exist only in A’s memory until they’re sent. When B then creates a file, the server recalls the directory from A, A sends all of its creates in one message and gives the directory back, and the server answers B. B waits for two round trips: its own and the recall.

The explorable also shows the cost. B’s first create after A took the directory waited for a recall as well as its own round trip. If A and B take turns creating files in the same directory, every turn is a recall, and that’s slower than if nobody had held the directory at all. Gray and Cheriton’s advice for leases applies here too: when another client is about to change the data, the cheapest grant is none. Holding a directory pays off when one client does most of the work in it, which is how a clone, an install, or a build behaves. For how other clients see the holder’s changes, see the next chapter, [One file, many caches](https://archil.com/guides/how-file-systems-work/one-file-many-caches).

## What still costs a round trip

Holding a directory removes the waits on what a client creates and changes. Three kinds of wait remain, whatever the design.

- **Reading something for the first time.** A client can answer from memory only what it has already fetched. A large tree that a job has never read costs a round trip per miss, unless the client fetches it ahead of time.
- **Making changes durable.** `fsync()` has to reach storage that will outlive the machine, so it waits for at least one round trip to wherever that storage is.
- **Taking the directory.** The first client to hold a directory asks for it once, and a client that wants a directory someone else holds waits for a recall.

These are the waits to count when you read a vendor’s description of its protocol. Ask what happens on a create, an open, and a close, and which of them wait for a reply.

## A worked example: Archil’s checkout

Archil, the file system we build, uses this approach. A client that holds part of the file system is said to have it _checked out_. Our docs and CLI call the grant a delegation, and the commands are `archil checkout` and `archil checkin`. (The modes and commands in this section are from [Sharing disks](https://docs.archil.com/concepts/sharing-disks) in Archil’s documentation.) A client mounts a disk in one of three modes:

- In the default exclusive mode, the client that mounts a disk holds all of it. It answers creates, writes, renames, and attribute changes from memory, and sends the changes to our servers in the background and on every `fsync()`. Other machines can mount the disk read-only.
- In shared mode, a client can read the whole disk, and it writes under paths it has checked out with `archil checkout <path>`. Creating a file in a directory nobody holds checks that file out automatically.
- In conditional mode, any number of clients write the same paths at once. Creates, writes, renames, and deletes run on our servers before they return, as they do over NFS, the last write wins, and the docs say performance falls to that of a remote file system. A client that does most of the writing in a path can still check it out, to write there at local speed while other clients read.

A checkout differs from a lease in one way that matters. It doesn’t expire. If a client crashes while holding a path, nobody can write there until someone runs `archil checkout <path> --force`, and that discards the crashed client’s writes that hadn’t reached our servers. Gray and Cheriton’s lease takes the other side of that trade: it waits out the term and loses nothing, at the cost of a delay after every failure. Writes that `fsync()` returned for are already on our servers, so a forced checkout loses at most what the program hadn’t synced.

For what these round trips cost agents that install packages and clone repositories, see [Storage for agents](https://archil.com/guides/storage-for-agents).