Storage for agentsChapter 3 of 6
Why it’s hard: where the truth lives
The short answer
Every storage system decides which machine may answer a read or a write without asking anyone. Speed wants that to be the machine doing the work, safety wants copies elsewhere before anything is acknowledged, and sharing wants a single authority. The design that satisfies all three keeps the truth remote and replicated and lends the right to answer to the working machine, and building it is a long list of hard problems.
Hunter LeathFounder & CEO, Archil
12 min readChecked November 2026Read as Markdown
An agent runs touch notes.md. Before that call returns, some machine has to decide that the name is free, record it, and answer. On a laptop, the kernel decides in its own memory and returns without waiting for anything. On a network file system, a server decides, and the agent waits for a round trip.
Which machine gets to decide is the most consequential choice a storage system makes. The previous chapter showed each familiar design making that choice once and living with the consequences. This chapter is about the choice itself, and about the old idea that gets around it.
Three demands on one answer§
Every read and every write needs a machine that’s allowed to answer it. Call the copy that machine answers from the authoritative copy: when two copies disagree, it’s the one that’s right. Three demands pull on where it lives.
Speed wants the answer here. A local file system answers open, write, chmod, and close from the kernel’s memory, and the disk catches up later. Any call that has to ask another machine waits at least one round trip instead. A job that creates thousands of files pays that wait thousands of times.
Safety wants copies elsewhere before anyone is told. In the cloud, losing a machine is routine, and a stop, a termination, or a host retirement erases the drives attached to the host.See The work outlives the machine, and the field guide’s Where fsync returns from. A write survives that only if it already exists on other machines, ideally in other availability zones. So fsync() has to wait for those copies, which costs at least one round trip every time a program asks for safety.
Sharing wants one authority. Suppose machines A and B both create hello.txt in the same directory. Only one of them can win, and neither knows about the other’s request. The only place that can decide is a place both requests reach, and every other machine has to ask it.
Three demands on one answer. Speed: machine A’s kernel answers open, write, fchmod, and close from memory, with no round trips. Safety: A’s fsync() sends the write to servers in other zones before it’s acknowledged, so when A is lost, the write survives. Sharing: machines A and B both create hello.txt, and only an authority that sees both requests can decide. A’s file is created, and B is told the file exists.
Each demand is reasonable alone, and together they conflict. Speed puts the authority on the machine doing the work. Sharing puts it in one place that every machine can reach. Safety says that, wherever it is, no answer goes out until the data is off any single machine.
Moving the authority§
The trade-off is easiest to see by moving the authority yourself. The following explorable has three places for it: the machine where the agent works, a server, and a server that lends the working machine a checkout. A checkout is the exclusive right to answer for part of the file system. The distances are round trips we measured: within one AWS region, and from sandboxes on two other providers’ platforms.
The create, write, chmod, and close are answered from memory. Only fsync() waits, for one trip to a log that’s replicated before it answers.
npm ci of 18,602 files waits for none of them. The changes travel in the background in batches, and a final sync waits once.
Synced writes are on the replicated log. Writes the program never synced, at most the last moment before a background send, are lost. The checkout stays with the lost machine until another machine takes it over.
The holder sends its changes about once a second and on every fsync(), and B sees them once its own cached copy expires. B can write there only after the holder hands the checkout back.
A server lends this machine a checkout, round trip 0.5 ms. Create: 1 round trip · 0.5 ms. npm ci of 18,602 files waits for none of them. The changes travel in the background in batches, and a final sync waits once. If A is lost: Everything fsync() acknowledged survives. Machine B sees: What the holder has sent.
An explorable with three positions for the authority, three measured server distances (0.5 ms within one region, about 10 ms from a Vercel sandbox, and about 40 ms from a Modal sandbox), and a button that loses machine A. Creating one file takes four calls: open with O_CREAT, write 4 KiB, fchmod, and close, followed by fsync. When this machine answers, the create and fsync wait for no round trips, and an npm ci of 18,602 files waits on no network. If A is lost, everything written since the last copy left it is lost, including writes fsync acknowledged, and machine B can never see the files. When a server answers, each create waits for four round trips (CREATE, SETATTR, and WRITE plus COMMIT on close): 2.0 ms within the region, 40 ms from the Vercel sandbox, and 160 ms from the Modal sandbox. The npm ci waits for 74,408 round trips: about 37 s, about 12 minutes, or about 50 minutes. Nothing acknowledged is lost with A, and B sees every change once its cached copy expires; Linux NFS keeps a directory’s attributes for 30 to 60 s by default. B can write too, and the server decides every conflict. When the server lends A a checkout, the create, write, chmod, and close are answered from memory, and only fsync waits, for one round trip to a replicated log: 0.5 ms, 10 ms, or 40 ms. The npm ci waits for none of its creates; changes travel in background batches, and a final sync waits once. If A is lost, everything fsync acknowledged survives, writes never synced are lost, and the checkout stays with the lost machine until another machine takes it over. B reads what A has sent, about once a second and on every fsync, once its own cache expires, and can write under A’s checkout only after A hands it back.
With the authority on this machine, every answer is local and nothing waits. That’s the sandbox’s own disk, and it’s also a disk that uploads to object storage in the background, because until the upload finishes, the only copy is here. Lose the machine and the recent work goes with it, acknowledged or not. No other machine can see the files at all.
With the authority on a server, every change asks. That’s NFS, and it’s Amazon EFS. It’s safe and it’s shared, and a create costs four round trips. Multiply that by the files in a clone or an install, and the network sets the wall-clock time, not the disk.
With the authority on a server that lends a checkout, the working machine answers like a local disk, and the truth stays on replicated storage elsewhere. Only fsync() waits, once. Measured systems keep the same order, though a real server adds its own work to each round trip. In our measurements, creating a 4 KiB file and syncing it took 0.18 ms on a Vercel Drive, which acknowledges from local NVMe. It took 8.5 ms on Amazon S3 Files, an NFS server in the same zone. On Archil, where the client holds a checkout and only the fsync() waited, it took 4.2 ms.Vercel Drive from a Vercel sandbox in pdx1; S3 Files and Archil from one c7gn.2xlarge in us-east-1. Medians. Methods are in our sandbox storage benchmarks and file system benchmarks.
The third position is the only one that does well on all three readouts. It’s also the one with the most moving parts, and the idea behind it is older than any product that uses it.
Lending the right to answer§
The Andrew File System, built at Carnegie Mellon, ran into this problem early and measured it carefully. John Howard and six colleagues described it in “Scale and Performance in a Distributed File System,” published in 1988. In the AFS prototype, each workstation checked with the server before using any cached file, and two kinds of status call made up nearly 90% of the calls its servers received. The revised AFS turned that around: “When a workstation caches a file or directory, the server promises to notify it before allowing a modification by any other workstation.” They called the promise a callback. On their benchmark, the prototype ran 70% slower than a workstation with a local disk, and the revised system 19% slower.
A callback has no end date, which is a problem when the workstation holding it crashes. In 1989, Cary Gray and David Cheriton put an expiry on the promise in “Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency”: “A lease is a contract that gives its holder specified rights over property for a limited period of time.” If a holder vanishes, the server waits out the term, and then lets the write go ahead. A failure costs time, and no client reads a stale copy.
NFS version 4 built both ideas into NFS: its state lives under leases, and the server recalls what it lent with a callback. Its first specification, RFC 3010 in December 2000, introduced “the ability of the server to delegate certain responsibilities to the client.” A client holding a write delegation opens, closes, and writes a file without asking, until the server recalls it. NFSv4.1, in RFC 5661 in 2010, added directory delegations, so a client could list a directory and look names up from memory.
What a client may answer without asking the server, by mechanism. AFS callbacks, 1988: reuse cached reads, open and close, and write a file’s data, but not create names; AFS sent directory changes to the server. Leases, 1989: reuse cached reads, and open and close if the lease covers the name, but writes and creates go to the server. NFSv4 write delegations, 2000: reads, opens, closes, and data writes, but not creates. NFSv4.1 directory delegations, 2010: cached reads of the directory only, because they’re read-only. A checkout of a directory tree: all four, including creating names.
Every one of these protocols stops short of creates. AFS made directory changes “directly on the server responsible for that directory.” An NFSv4.1 directory delegation is “read-only,” in RFC 8881’s words, so a client holding one still sends every create to the server.Both quotations, and the rest of this lineage, are worked through in the field guide’s Anatomy of a file create and One file, many caches. An agent’s work is mostly creates: a clone, an install, a build. The last row of the figure is the step it needs.
Truth remote, answers local§
The design that follows from this history keeps the authoritative copy remote, in a log replicated across availability zones, and lends the right to answer to the machine doing the work. While machine A holds a checkout of /repo, nobody else may change anything under it. So A can answer creates, writes, mode changes, and renames from memory, exactly as a laptop does.
The changes still travel, but not one at a time. A sends them to the log in batches in the background, and immediately when a program calls fsync(). That fsync() returns once the log has stored the changes on other machines, so an acknowledged write never depends on A alone. When machine B wants to write under /repo, it asks the server. The server recalls the checkout, A sends everything it still holds and hands the checkout back, and then B holds it.
Machine A checks out /repo from the server. It creates a.js and b.js, changes a mode, and renames a file, each answered in its own memory with no round trips. fsync() sends the four changes in one batch to a log that’s replicated across zones before it acknowledges, which takes one round trip. A then creates c.js, which isn’t synced yet. Machine B asks to write under /repo, so the server recalls the checkout. A sends c.js and hands the checkout back, and B holds /repo and answers locally.
Each demand gets what it asked for. Speed gets local answers for the machine doing the work. Safety gets every acknowledged write on replicated storage off that machine. Sharing gets one authority per path at every moment, and the server always knows which machine it is.
The design has costs, and they’re worth knowing before you rely on it.
- Contention. If two machines take turns writing in the same directory, every turn is a recall, and that’s slower than asking the server each time. A checkout pays off when one machine does most of the work in a path, which is how a clone, an install, or a build behaves.
- A holder that vanishes. A lease waits out its term. A checkout without an expiry has to be taken over by force, and the writes the old holder never synced are gone. Either way, a lost machine costs something.
- First reads. A machine can answer from memory only what it has already fetched. Reading a large tree for the first time still costs a round trip per miss.
This is the design Archil is built on. Our docs call the grant a delegation, and the commands are archil checkout and archil checkin. What Archil does, and where it falls short covers our implementation and its limits.
What building it takes§
I spent eight years on the Amazon EFS team. The diagram in the previous section is the part of this work that fits on a whiteboard. The following list is the part that doesn’t, and each item is a project of its own.
An iceberg. Above the waterline: mounting a disk, reading, and writing, which is what a demo shows. Below it, eight pieces of work: handing authority over, a log replicated across zones before fsync() returns, recovery after a crash, the corners of POSIX, cache invalidation, copy-on-write metadata, syncing to a bucket in its native format, and isolation between tenants.
A protocol for handing authority over. Granting a checkout is the straightforward step. A recall has to collect every change from the holder before the server answers the next machine, or B lists a directory that’s missing A’s files. A holder that stops answering forces a choice between waiting, which needs clocks with bounded drift, and taking over, which loses its unsynced writes. The storage also has to refuse writes from a holder that lost its grant, because a lock the storage never checks protects nobody. The field guide covers handing a directory over and why locks across machines are hard.
A log replicated across zones before fsync() returns. Every acknowledged write crosses at least one zone boundary, so the log has to batch and append to keep fsync() near a single round trip. It also can’t wait for its slowest copy, or one slow server sets the pace for every disk on it. A quorum waits for enough copies instead of all of them. See Quorums.
Recovering half-flushed state after a crash. The log is eventually folded into cheaper storage, and a server can crash halfway through. Some new objects are written, and the pointer that would make them live hasn’t moved. On restart, the server has to replay the log from the last point it knows is complete, and treat the half-written objects as garbage, not data. See Write-ahead logs and crash consistency in a copy-on-write tree.
The corners of POSIX. Tools rely on them without saying so. git updates its index by renaming index.lock over index, and needs the swap to be atomic. A rename across directories must check that a directory isn’t being moved inside itself, which Linux does under one lock per file system. Across machines, the authority has to play that lock. A hard link is two names for one file. A file that’s unlinked while a process still has it open keeps its data until the last close, which the NFSv3 specification says “is impossible for a stateless server to implement.” See Rename and Links, and the last close.
Cache invalidation. Every machine that cached a name, an attribute, or a page has to find out when it’s stale, by asking on a timer or by being told. Asking costs round trips and still leaves a window. Being told means the server tracks every copy and reaches each one before a write goes ahead, including copies on machines that have crashed. See Finding out a copy is stale.
Copy-on-write metadata. Forking a workspace in milliseconds, whatever its size, means the fork shares every block with its parent and stores only what changes. Then every write allocates new blocks and rewrites the path above them. Freeing a block requires knowing that no checkpoint or branch still points at it. See Deciding when a block is free, and Copies versus references for why it matters to agents.
Syncing to a bucket in its native format. If your files should land in your bucket as ordinary objects, each file operation needs an object-store translation, and some have none. A directory rename becomes a copy and a delete per object, and a reader partway through sees both directories. An object has no mode bits or owner of its own, and a hard link becomes two separate objects. See What an object can’t hold and Rename is a copy and a delete.
Isolation between tenants. A service shares its servers among many disks, which is what makes storage cheap when most of those disks sit idle. Then one tenant’s install of tens of thousands of files must not slow another tenant’s fsync(). A bug triggered by one disk’s state must not take down the server for every other disk on it. See Reserved capacity and consumed capacity for why services pool.
Two of these items, copy-on-write metadata and the bucket’s format, also decide what it costs to fork a workspace, hand it to another machine, or leave the vendor that stores it. Copies versus references takes those three moments in turn.