The obvious ways to build it, and where each one breaks

Every team that gives its agents a workspace starts with the storage it already has. Most then climb the same ladder. Each rung is a sensible answer to the rung before it, and each breaks somewhere new.

To compare the rungs fairly, this chapter runs one day through each of them. An agent clones a repository and runs npm ci. It commits, fsync() returns, and then the host it ran on is lost. A sandbox on a second machine picks up the work, forks three attempts at the next change, and then the workspace sits untouched for a week. The work outlives the machine explains why that day is ordinary for an agent.

Where we have measurements, the numbers are ours, and each section says what we measured on. Where we don’t, the chapter says what the vendor documents, and doesn’t guess.

The sandbox’s own disk§

The first workspace is the disk the sandbox came with. It’s the best storage on the ladder for the first half of the morning. Every tool works, every call is answered by the kernel on the same machine, and nothing later on the ladder is faster. Every other rung is measured against it.

It fails at the day’s third step. A sandbox runs on whichever host has room, so its disk is that host’s disk. Data on the host’s own drives doesn’t survive a retirement, a stop, or a termination.From AWS’s page on instance store data persistence. The work outlives the machine covers how hosts are retired. The clone, the install, and the commit go with it, and the sandbox that replaces it starts on an empty disk.

Even without a failure, the disk lasts only while the sandbox runs. An idle sandbox with 2 vCPUs and 4 GiB costs about $0.17 an hour at E2B’s list price.E2B bills $0.000014 per vCPU-second and $0.0000045 per GiB-second (E2B pricing). Keeping one up through a week of idling, so that its disk survives, costs about as much as storing a 5 GiB workspace in S3 Standard for 20 years.

An upload when the agent stops§

The second rung keeps the speed and adds a copy. The agent still works on the sandbox’s disk. The workspace goes to S3 at the end of each session as a tarball, or every few minutes with aws s3 sync or rclone. S3 stores objects across at least three availability zones, and a week of idling costs almost nothing.From AWS’s page on S3 data protection. Products sold to agent builders polish this design. Fly.io writes a Sprite’s data “to fast NVMe storage” and backs it up “to durable object storage” when the Sprite goes idle. Vercel describes its Drives as having “NVMe speed writes.”From the Sprites overview and Vercel’s Drives documentation.

The failure is the gap between a write and its upload. fsync() returns once the local drive has the data, and the host can be lost before the copy is made. We measured that gap on Vercel Drives. In 21 of 21 snapshots taken the moment fsync() returned, the file wasn’t there.

Drives and Sprites are reasonable trades for speed. But a database that tells a user their transaction committed has called fsync() and believed the answer. For more on where that answer comes from, see When is a write safe?

The rest of the day costs copies. The second machine downloads the last upload before it can start, and loses whatever came after it. Three attempts mean three downloads. A tarball keeps modes and links but moves the whole workspace every time. A sync tool moves only what changed, but as one object per file, which can’t hold everything a file can, as A file system on an object store explains.

Every team that builds this rung patches it, and each patch is sensible. The following list shows where the common ones lead:

• Upload more often. Syncing every minute instead of at the end shrinks the gap, but it never closes it. A busy host always has writes in flight when it dies, and the program was already told they were safe. In our Drive test, 7 of 10 snapshots taken 30 seconds after fsync() were still missing the file.
• Leave out node_modules and the build output. The uploads shrink, and every new machine reinstalls before it can work. The npm ci in our measurements took 10.7 s on a local disk, paid again on every start.
• Snapshot with tar instead of syncing. A snapshot is consistent and keeps modes and links, but it moves the whole workspace each time. So it runs less often, and the gap grows.
• Make the sync safe to run. rclone sync makes the destination match the source, “including deleting files if necessary,” and its documentation warns that “this can cause data loss.” If a restore fails quietly and the next upload runs from a nearly empty workspace, the good copy is what gets deleted.From the rclone sync documentation.
• Lock the workspace so that two machines can’t both write it. Now you run a lock service, and you have to decide what happens to a lock whose holder’s host was retired. Chapter 3 is about that decision.

Follow the patches far enough and they meet at one requirement. The copy has to be made before fsync() returns, on every write, and the next machine has to start without a download. That isn’t a sync job anymore. It’s a file system whose storage lives somewhere else, which is the next rung.

A bucket mounted as a file system§

If the gap is the problem, remove it and write straight to the bucket. Mountpoint for Amazon S3 and s3fs mount a bucket so that each file is an object. Mountpoint uploads a file before fsync() returns, so a lost host takes nothing with it. Any machine can mount the same bucket, and every file is an ordinary object that other tools can read. In our measurements, a second machine saw a new file in under a third of a second without being told to look.

It fails at the first step of the day. S3 can’t rename an object, and git depends on renames. It publishes every change by writing a lock file and renaming it over the original. npm links package binaries with symbolic links. Mountpoint supports neither rename on a general purpose bucket nor symbolic links, and its README says not to use it for a Git repository. s3fs fakes a rename with a copy and a delete, so git runs, but its README lists atomic renames among what it can’t provide.From the Mountpoint README, its semantics document, and the s3fs README.

Where it does work, every small file pays for a round trip to S3. Creating a 4 KiB file and syncing it took 244 ms on Mountpoint on our EC2 host. A mounted bucket suits training data that is written once and read many times. An agent’s workspace is the opposite.

A network volume§

The fourth rung goes back to a real disk, one that lives on the network. To the kernel, an EBS volume is a block device with ext4 on it, so git, npm, and SQLite run at local-disk speed. The local disk in our benchmarks is in fact an EBS gp3 volume. And the volume outlives the machine: “EBS volumes persist independently from the running life of an EC2 instance,” and EBS replicates each one within its availability zone.From the AWS pages on Amazon EBS volumes and EBS features. It’s the first rung that runs the whole morning and keeps the commit.

It breaks at the handoff. The file system on the volume keeps its map of blocks in one kernel’s memory, so the volume attaches to one machine at a time. And “the volume and instance must be in the same Availability Zone.” Taking it from a failed instance means a forced detach, which AWS calls a “last resort.” The instance “doesn’t get an opportunity to flush file system caches or file system metadata,” so you check the file system before you use it.From the AWS pages on attaching and detaching a volume. Multi-Attach lets an io1 or io2 volume attach to several instances in one zone, but ext4 and XFS aren’t built for that (Multi-Attach).

The rest of the day follows from the same fact. Three attempts mean a snapshot and three new volumes, and a volume restored from a snapshot fetches its blocks from S3 as they’re read, so it starts slow.From AWS’s Amazon EBS snapshots and Initialize Amazon EBS volumes pages. And a volume bills the size you gave it, idle or not. A 20 GiB volume holding a 5 GiB workspace bills four times what it holds, and EBS can grow a volume but can’t shrink it.

A shared file system over NFS§

The fifth rung lets every machine mount the same files at once. On Amazon EFS, any number of machines mount one file system over NFS. A Regional file system stores a write across availability zones when the program calls fsync() or closes the file, and you pay for what you store.From the AWS page on EFS features. The host can be lost, and the second machine mounts the file system and finds the commit. It’s the first rung that runs the morning’s tools and gets through the middle of the day without a forced detach or a download.

It breaks on the morning’s small files. When two machines create the same name, only the server can decide which one wins. So an NFS client waits for the server on every create, every change of mode, and every close. A small file costs about four round trips. For how those calls add up, see Anatomy of a file create. Amazon S3 Files, which serves a bucket over NFS, shows the cost on the install.

The install takes almost seven times as long over NFS as on the volume. EFS itself took 86.3 s for npm ci of a create-next-app project in a separate benchmark, 11 times as long as Archil in the same run.That benchmark stopped the clock when each command returned. For the hosts, versions, and commands behind both benchmarks, see File systems on one host. NFS clients also cache what they’ve seen, so a second machine sees a new file only when its cached copy of the directory expires. Archil’s clients do the same by default.

Forks are the last cost on this rung. EFS has no clone, so each attempt copies every file over NFS and pays the round trips of the install again.

JuiceFS, and a database you run§

The sixth rung stops asking one system to do everything. JuiceFS keeps names, directories, modes, links, and the map from each file to its blocks in a metadata database that you run, such as Redis or PostgreSQL. It keeps file contents as numbered blocks in your bucket.From the JuiceFS architecture page. The Community Edition is Apache 2.0. A rename changes one row, a write in the middle of a file uploads only the new bytes, and any machine can mount it. juicefs clone copies only metadata, so forks share blocks. It’s the most complete of the do-it-yourself designs.

The cost is the database, which you now operate, and the file system is only as safe as it is. JuiceFS’s own guide to Redis says the default log sync means “you may lose the writes from the last second.” It also says failover “does not guarantee that acknowledged writes are retained during failures,” and “Redis data needs to be backed up regularly.”From JuiceFS’s Redis best practices. Lose the database, and JuiceFS’s docs say “you cannot find the original file directly in the object storage.”

The database also sits in the path of every small file. With Redis on ElastiCache in the same availability zone, our npm ci took 161.6 s with JuiceFS’s defaults, 15 times as long as on the volume. Its write-back mode is faster because it stages blocks on local disk and uploads them later. Its docs warn that “if write cache data suffers loss before upload is complete, file data is lost forever.”From JuiceFS’s guide to cache. Our setup is in the file system benchmark. That is the second rung’s gap, back again.

One day on every rung§

Run the day yourself. Pick a design, and each step shows whether the workspace comes through it and how long it took where we measured it. The idle week is priced at list price for a workspace holding 5 GiB. Where we have no measurement, the step says what the vendor documents instead.

Every rung gets through part of the day. The fast ones lose the host or stay with one machine in one zone. The ones that share are slow on small files, can’t run git, or need a database you run.

Where the authoritative copy lives§

Each rung failed for its own reasons, but the reasons share a shape. Each design decides where the authoritative copy of the workspace lives, and which machine may answer a read or a write without asking anyone else. The following figure puts each rung where it keeps that copy.

A design that fixes the authority in one place gets part of the day, and climbing to the next rung only moves the authority somewhere else. The way off the ladder is to stop fixing it in one place: keep the truth remote and replicated, and lend the right to answer to whichever machine is doing the work. The idea is decades old, and Where the truth lives covers why it works and what building it takes.