Anatomy of a file create over a network

Creating a file looks like one action, and on a laptop it costs about as much as one. Over a network it can cost several round trips, and a job that creates thousands of files pays that cost thousands of times. This page builds up why, one step at a time. It starts with a create on a single machine, sends the same create to a server, and then follows the three ideas that let a client stop asking, which are callbacks, leases, and delegations.

One file on one machine§

A program that unpacks a package creates each file with a handful of system calls:

fd = open("node_modules/react/index.js", O_CREAT | O_WRONLY, 0644);
write(fd, buf, 4096);
fchmod(fd, 0644);
close(fd);

Each call changes a different part of the file system. open with O_CREAT adds a name to a directory and allocates an inode, the record that holds a file’s owner, mode, size, and timestamps. For more information about inodes and directories, see From a path to an inode. write puts 4 KiB of data into the file. fchmod changes the mode in the inode. close releases the descriptor.

On Linux, none of these calls waits for the disk. The kernel keeps directory entries, inodes, and file data in memory: the dentry cache, the inode cache, and the page cache. Each call changes memory and returns. The disk catches up later. ext4, for example, commits its journal every 5 seconds by default, and data reaches the disk when the kernel writes back dirty pages, or when a program calls fsync(). Where fsync returns from follows that path in detail.

This works because one kernel owns this file system, so it’s the only place that can change a name in that directory. It never has to ask anyone whether index.js already exists.

The same create, sent to a server§

Sun’s Network File System puts a file system behind a server, to give many machines “transparent remote access to shared files across networks.” The specification of its second version, published as RFC 1094 in March 1989, states a goal that shaped everything after it: “The NFS protocol was intended to be as stateless as possible.” A stateless server keeps no record of its clients. If it crashes and restarts, clients retry their requests and carry on, with no state to rebuild.

The NFS client in the kernel turns each system call into a remote procedure call. NFS version 3, specified in RFC 1813, has a procedure for each kind of change, and the small file in the preceding section needs four of them:

• CREATE makes the name and the inode.
• SETATTR sets the mode that fchmod asked for.
• WRITE sends the 4 KiB of data. NFSv3 lets the server hold this write in memory, marked UNSTABLE.
• COMMIT asks the server to put that data on stable storage. Clients send the pending write and the commit when the file is closed. RFC 7530 makes that a rule for NFSv4 and notes that many NFSv3 clients already follow it.

CREATE and COMMIT are on RFC 1813’s list of synchronous procedures: the server writes the change to stable storage before it replies. So each round trip includes a durable write on the server, as well as the network.

The cost of a file is now the number of calls that wait, multiplied by the distance to the server. The following explorable does that multiplication. Move the server closer or farther away, and choose a job.

Real clients overlap some of this work. A package manager runs several unpacks at once, and the kernel can have several requests in flight. But each file’s own calls happen in order, and a job of thousands of files still pays thousands of waits.

Why only the server can decide§

Suppose two machines, A and B, both run touch hello.txt in the same shared directory at the same moment. Only one of them can create the file. A’s kernel doesn’t know about B’s request, and B’s kernel doesn’t know about A’s. The server is the only place where both requests arrive, so it’s the only place that can decide.

NFS gives the server the tools to decide. A CREATE request can be GUARDED, which tells the server to check for a file with the same name first and fail with NFS3ERR_EXIST if one is there. It can also be EXCLUSIVE, which handles a retried request correctly by storing a verifier with the new file. RFC 1813 adds that after an exclusive create, the client “must issue a SETATTR to set the correct file attributes,” which is one reason SETATTR follows CREATE so often.

The second half of that story is the expensive half. A goes on to create files that no other machine will ever touch, and every create still crosses the network. A stateless server keeps no record of who is working in which directory, so it can’t tell A that it has the directory to itself. A can’t know that either. Asking is the only safe choice, so A asks every time.

What a client may cache§

A client can’t decide a create alone, but it can remember what it has read. NFS clients cache file attributes and directory contents, and treat them as current for a while. Linux’s defaults, from the nfs(5) man page, are 3 to 60 seconds for a file’s attributes and 30 to 60 seconds for a directory’s.

Caching raises the question of when one client sees another’s change. NFS clients answer it with close-to-open consistency. When a program closes a file, the client sends its pending writes to the server. When a program opens a file, the client asks the server for the file’s current attributes, “regardless of the freshness of the file’s cached attributes,” in the man page’s words. If A closes a file before B opens it, B sees A’s changes.

This is a good fit for files that one machine writes and another reads later, which is most sharing. It doesn’t make anything faster for a create: the close still waits for the server, and the next open still asks. It also leaves a window in which a listing is stale. The Andrew File System paper described NFS as a system “in which a file created on a workstation may not be visible on another workstation for 30 seconds.”

Callbacks: the Andrew File System§

The Andrew File System was built at Carnegie Mellon, which planned for it to span more than 5,000 workstations. John Howard and six colleagues described its first two versions in “Scale and Performance in a Distributed File System,” published in ACM Transactions on Computer Systems in February 1988. The paper is a careful record of a design that didn’t scale, the measurements that showed why, and the change that fixed it.

AFS caches whole files on each workstation’s local disk. A process called Venus manages the cache and talks to the servers, called Vice. In the prototype, Venus treated every cached file as suspect: “Before using a cached file, Venus would verify its timestamp with that on the server responsible for the file. Each open of a file thus resulted in at least one interaction with a server, even if the file were already in the cache and up to date.” Programs that called stat before opening a file made it worse, because each stat checked the cache too.

Two calls made up nearly 90% of the traffic to the servers: TestAuth, which validated a cache entry, and GetFileStat, which fetched the status of a file absent from the cache. Only 6% of calls moved file data. Performance was usually acceptable up to about 20 active users per server. In the team’s benchmark, TestAuth response times rose rapidly once more than five workstations ran it at the same time.

The revised AFS turned the question around. In the paper’s words, “When a workstation caches a file or directory, the server promises to notify it before allowing a modification by any other workstation. This promise, called a Callback, dramatically reduces the number of cache validation requests received by servers.” Venus now assumes a cached entry is valid unless told otherwise, so an open of a cached file needs no message at all.

The numbers moved a long way. The prototype ran the benchmark 70% slower than a workstation with a local disk, and the revised system ran it 19% slower. In the prototype, the benchmark took more than four times as long at a load of 10 as at a load of 1. In the revised system it took 36% longer at that load, and less than twice as long at a load of 20. The authors concluded that their goal of 50 users per server had been met. Callbacks also let workstations resolve path names from their caches, because a cached directory with a callback on it can be trusted.

Callbacks cost something too. The server now keeps state for every cached file, and must notify every holder before it allows a change. A rebooted workstation can’t trust anything in its cache and has to revalidate each entry the first time it’s used. And callbacks covered reading, not writing names: “For reasons of integrity, modifications to a directory are made directly on the server responsible for that directory.” In AFS, a create still went to the server.

Leases: a promise with an expiry§

A callback is a promise with no end date, which is a problem when machines fail. If the server can’t reach a workstation to break a callback, it can wait forever, or go ahead and let the workstation read a stale copy. In 1989, Cary Gray and David Cheriton at Stanford proposed a way out in “Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency,” published in the proceedings of the 12th ACM Symposium on Operating Systems Principles.

Their definition is short: “A lease is a contract that gives its holder specified rights over property for a limited period of time.” A cache that holds a lease on a file can serve reads of it without asking the server. When another client wants to write, the server must get the leaseholder’s approval, or wait until the lease has expired. A lease can cover the file’s name and permissions, not only its contents, so a rename counts as a write.

The expiry is what makes leases tolerate failure. If a leaseholder crashes or the network splits, the server waits out the lease and then lets the write proceed, knowing nobody can still be relying on the old copy. A failure costs the writer time, and no client reads a stale copy. A server that restarts only has to remember the longest lease it granted, and hold writes for that long.

The paper then asks how long a lease should last, and the answer is shorter than you might expect. In their model, built on file access data from Stanford’s V system, a 10-second term on a file cached by one client cut the traffic spent on consistency to 10% of what a zero-length term needed. Longer terms added little. Gray and Cheriton placed AFS on the same scale: its prototype amounted to a lease term of zero, and its revised version to a term of infinity. Short terms also limit false sharing, where a writer has to wait for a lease that’s held by a client no longer using the file.

Delegations in NFSv4§

NFS version 4 gave up statelessness. Its specification, now RFC 7530, keeps a client’s opens, locks, and other state on the server under a single lease that the client renews. On that foundation it adds delegations, a mechanism “that allows many decisions normally made by the server to be made locally by clients.”

An open delegation hands a client the right to handle opens and closes of one file by itself. With a read delegation, other clients may not change the file. With a write delegation, the client may change the file’s data, and only the attributes tied to it: size, modification time, and the change attribute. While it holds either, the client “does not send OPENs or CLOSEs to the server.” When another client wants the file, the server recalls the delegation with a callback, as AFS did. The holder sends its modified data back and returns the delegation, and the second client’s request waits until it does.

NFSv4.1, now RFC 8881, added directory delegations for exactly the case on this page. The RFC names compilation as a workload with many lookups of files that don’t exist yet, and credits AFS and DFS with showing that state about directory contents “can greatly reduce network traffic.” Directory delegations are optional for servers, and they stop short of writing: “The delegation is read-only and the client may not make changes to the directory other than by performing NFSv4.1 operations that modify the directory or the associated file attributes so that the server has knowledge of these changes.” A client with a directory delegation can list the directory and miss on lookups from memory. It still sends every create to the server.

The following table summarizes what each mechanism lets a client do without asking the server.

Answering a create from memory§

The last row is what a small-file job needs. For a client to answer open(O_CREAT) by itself, the server has to hand it the directory: the exclusive right to add and remove names there. Everything else follows the patterns this page has already built.

• The right has to be exclusive, like a write delegation, because two clients deciding names in one directory is the conflict the server exists to prevent.
• The server needs a way to take it back. When another client asks for the directory, the server recalls it, the holder sends every change it made, and only then does the server answer the second client.
• The changes still travel, but not one at a time. A client that created 500 files sends them to the server together, so 500 waits become one.
• Durability is a separate question. A create that exists only in the holder’s memory is not yet safe, exactly as a write on a local disk isn’t safe before fsync(). For what has to happen before fsync() returns, see Where fsync returns from.

Try it in the following explorable. Create a few files from A while nobody holds the directory, then hand A the directory and create a few more. Then create a file from B.

The explorable also shows the cost. B’s first create after A took the directory waited for a recall as well as its own round trip. If A and B take turns creating files in the same directory, every turn is a recall, and that’s slower than if nobody had held the directory at all. Gray and Cheriton’s advice for leases applies here too: when another client is about to change the data, the cheapest grant is none. Holding a directory pays off when one client does most of the work in it, which is how a clone, an install, or a build behaves. For how other clients see the holder’s changes, see the next chapter, One file, many caches.

What still costs a round trip§

Holding a directory removes the waits on what a client creates and changes. Three kinds of wait remain, whatever the design.

• Reading something for the first time. A client can answer from memory only what it has already fetched. A large tree that a job has never read costs a round trip per miss, unless the client fetches it ahead of time.
• Making changes durable. fsync() has to reach storage that will outlive the machine, so it waits for at least one round trip to wherever that storage is.
• Taking the directory. The first client to hold a directory asks for it once, and a client that wants a directory someone else holds waits for a recall.

These are the waits to count when you read a vendor’s description of its protocol. Ask what happens on a create, an open, and a close, and which of them wait for a reply.

A worked example: Archil’s checkout§

Archil, the file system we build, uses this approach. A client that holds part of the file system is said to have it checked out. Our docs and CLI call the grant a delegation, and the commands are archil checkout and archil checkin.The modes and commands in this section are from Sharing disks in Archil’s documentation. A client mounts a disk in one of three modes:

• In the default exclusive mode, the client that mounts a disk holds all of it. It answers creates, writes, renames, and attribute changes from memory, and sends the changes to our servers in the background and on every fsync(). Other machines can mount the disk read-only.
• In shared mode, a client can read the whole disk, and it writes under paths it has checked out with archil checkout <path>. Creating a file in a directory nobody holds checks that file out automatically.
• In conditional mode, any number of clients write the same paths at once. Creates, writes, renames, and deletes run on our servers before they return, as they do over NFS, the last write wins, and the docs say performance falls to that of a remote file system. A client that does most of the writing in a path can still check it out, to write there at local speed while other clients read.

A checkout differs from a lease in one way that matters. It doesn’t expire. If a client crashes while holding a path, nobody can write there until someone runs archil checkout <path> --force, and that discards the crashed client’s writes that hadn’t reached our servers. Gray and Cheriton’s lease takes the other side of that trade: it waits out the term and loses nothing, at the cost of a delay after every failure. Writes that fsync() returned for are already on our servers, so a forced checkout loses at most what the program hadn’t synced.

For what these round trips cost agents that install packages and clone repositories, see Storage for agents.