# 2. The obvious ways to build it, and where each one breaks

Source: https://archil.com/guides/storage-for-agents/the-obvious-ways-to-build-it · Part of The buyer’s guide to storage for AI agents · Facts checked 2026-11-04

**The short answer:** Teams that build storage for agents themselves tend to climb the same ladder, from the sandbox’s own disk to uploads to S3, a mounted bucket, a network volume, NFS, and JuiceFS. Each rung fixes the failure of the one before it and meets a new one. Each puts the authoritative copy of the workspace in one place, and that place decides what breaks.

Every team that gives its agents a workspace starts with the storage it already has. Most then climb the same ladder. Each rung is a sensible answer to the rung before it, and each breaks somewhere new.

To compare the rungs fairly, this chapter runs one day through each of them. An agent clones a repository and runs `npm ci`. It commits, `fsync()` returns, and then the host it ran on is lost. A sandbox on a second machine picks up the work, forks three attempts at the next change, and then the workspace sits untouched for a week. [The work outlives the machine](https://archil.com/guides/storage-for-agents/the-work-outlives-the-machine) explains why that day is ordinary for an agent.

Where we have measurements, the numbers are ours, and each section says what we measured on. Where we don’t, the chapter says what the vendor documents, and doesn’t guess.

## The sandbox’s own disk

The first workspace is the disk the sandbox came with. It’s the best storage on the ladder for the first half of the morning. Every tool works, every call is answered by the kernel on the same machine, and nothing later on the ladder is faster. Every other rung is measured against it.

It fails at the day’s third step. A sandbox runs on whichever host has room, so its disk is that host’s disk. Data on the host’s own drives doesn’t survive a retirement, a stop, or a termination. (From AWS’s page on [instance store data persistence](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-store-lifetime.html). [The work outlives the machine](https://archil.com/guides/storage-for-agents/the-work-outlives-the-machine#a-machine-built-to-be-thrown-away) covers how hosts are retired.) The clone, the install, and the commit go with it, and the sandbox that replaces it starts on an empty disk.

**The disk goes with the host**

A sandbox on host 1 clones a repository, installs its packages, and commits, all on its own disk. Host 1 is retired, and the disk goes with it. The sandbox starts again on host 2 with an empty disk, so the clone, the install, and the commit are gone.

Even without a failure, the disk lasts only while the sandbox runs. An idle sandbox with 2 vCPUs and 4 GiB costs about $0.17 an hour at E2B’s list price. (E2B bills $0.000014 per vCPU-second and $0.0000045 per GiB-second ([E2B pricing](https://e2b.dev/pricing)).) Keeping one up through a week of idling, so that its disk survives, costs about as much as storing a 5 GiB workspace in S3 Standard for 20 years.

## An upload when the agent stops

The second rung keeps the speed and adds a copy. The agent still works on the sandbox’s disk. The workspace goes to S3 at the end of each session as a tarball, or every few minutes with `aws s3 sync` or rclone. S3 stores objects across at least three availability zones, and a week of idling costs almost nothing. (From AWS’s page on [S3 data protection](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html).) Products sold to agent builders polish this design. Fly.io writes a Sprite’s data “to fast NVMe storage” and backs it up “to durable object storage” when the Sprite goes idle. Vercel describes its Drives as having “NVMe speed writes.” (From the [Sprites overview](https://docs.fly.io/sprites/) and Vercel’s [Drives documentation](https://vercel.com/docs/sandbox/concepts/drives).)

The failure is the gap between a write and its upload. `fsync()` returns once the local drive has the data, and the host can be lost before the copy is made. We measured that gap on Vercel Drives. In 21 of 21 snapshots taken the moment `fsync()` returned, the file wasn’t there.

**A Vercel Drive after fsync returns**

A sandbox wrote a 4 KiB file and a 64 MiB file, called `fsync()` on each, and synced the directory. After a delay, a new sandbox read the Drive through a snapshot or a fork. Taken the moment `fsync()` returned, 10 of 10 snapshots, 11 of 11 snapshots taken while the writer kept writing, and 11 of 11 forks were missing the file. At 1 second, every snapshot and fork still lacked it. At 30 seconds, 7 of 10 snapshots and 9 of 10 forks lacked it, and 4 of 10 snapshots taken while the writer kept writing held a 64 MiB file whose last 12 to 28 MiB read as zeros. From 60 seconds on, the file was intact in every trial but one. An Archil disk mounted in a new sandbox the moment `fsync()` returned had the file in 20 of 20 trials.

Drives and Sprites are reasonable trades for speed. But a database that tells a user their transaction committed has called `fsync()` and believed the answer. For more on where that answer comes from, see [When is a write safe?](https://archil.com/guides/evaluating-storage/when-is-a-write-safe)

The rest of the day costs copies. The second machine downloads the last upload before it can start, and loses whatever came after it. Three attempts mean three downloads. A tarball keeps modes and links but moves the whole workspace every time. A sync tool moves only what changed, but as one object per file, which can’t hold everything a file can, as [A file system on an object store](https://archil.com/guides/how-file-systems-work/a-file-system-on-an-object-store) explains.

Every team that builds this rung patches it, and each patch is sensible. The following list shows where the common ones lead:

- **Upload more often.** Syncing every minute instead of at the end shrinks the gap, but it never closes it. A busy host always has writes in flight when it dies, and the program was already told they were safe. In our Drive test, 7 of 10 snapshots taken 30 seconds after `fsync()` were still missing the file.
- **Leave out `node_modules` and the build output.** The uploads shrink, and every new machine reinstalls before it can work. The `npm ci` in our measurements took 10.7 s on a local disk, paid again on every start.
- **Snapshot with `tar` instead of syncing.** A snapshot is consistent and keeps modes and links, but it moves the whole workspace each time. So it runs less often, and the gap grows.
- **Make the sync safe to run.** `rclone sync` makes the destination match the source, “including deleting files if necessary,” and its documentation warns that “this can cause data loss.” If a restore fails quietly and the next upload runs from a nearly empty workspace, the good copy is what gets deleted. (From the [rclone sync documentation](https://rclone.org/commands/rclone_sync/).)
- **Lock the workspace so that two machines can’t both write it.** Now you run a lock service, and you have to decide what happens to a lock whose holder’s host was retired. [Chapter 3](https://archil.com/guides/storage-for-agents/where-the-truth-lives) is about that decision.

Follow the patches far enough and they meet at one requirement. The copy has to be made before `fsync()` returns, on every write, and the next machine has to start without a download. That isn’t a sync job anymore. It’s a file system whose storage lives somewhere else, which is the next rung.

## A bucket mounted as a file system

If the gap is the problem, remove it and write straight to the bucket. Mountpoint for Amazon S3 and s3fs mount a bucket so that each file is an object. Mountpoint uploads a file before `fsync()` returns, so a lost host takes nothing with it. Any machine can mount the same bucket, and every file is an ordinary object that other tools can read. In our measurements, a second machine saw a new file in under a third of a second without being told to look.

It fails at the first step of the day. S3 can’t rename an object, and git depends on renames. It publishes every change by writing a lock file and renaming it over the original. npm links package binaries with symbolic links. Mountpoint supports neither rename on a general purpose bucket nor symbolic links, and its README says not to use it for a Git repository. s3fs fakes a rename with a copy and a delete, so git runs, but its README lists atomic renames among what it can’t provide. (From the [Mountpoint README](https://github.com/awslabs/mountpoint-s3), its [semantics document](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md), and the [s3fs README](https://github.com/s3fs-fuse/s3fs-fuse).)

**Five calls, five failures**

Five commands and the call each one depends on. `git commit` renames `index.lock` over `index`, and without an atomic rename git can’t update its index. SQLite writes a 4 KiB page at an offset in the database file, and without in-place writes each commit rewrites the whole file. `npm install` creates symbolic links in `node_modules/.bin`, and without symlinks the install fails. `pnpm install` hard-links files from its store into `node_modules`, and without hard links every file is copied. `git checkout` sets the executable bit on `build.sh`, and without mode bits `./build.sh` fails with `Permission denied`.

Where it does work, every small file pays for a round trip to S3. Creating a 4 KiB file and syncing it took 244 ms on Mountpoint on our EC2 host. A mounted bucket suits training data that is written once and read many times. An agent’s workspace is the opposite.

## A network volume

The fourth rung goes back to a real disk, one that lives on the network. To the kernel, an EBS volume is a block device with ext4 on it, so git, npm, and SQLite run at local-disk speed. The local disk in our benchmarks is in fact an EBS gp3 volume. And the volume outlives the machine: “EBS volumes persist independently from the running life of an EC2 instance,” and EBS replicates each one within its availability zone. (From the AWS pages on [Amazon EBS volumes](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes.html) and [EBS features](https://docs.aws.amazon.com/ebs/latest/userguide/EBSFeatures.html).) It’s the first rung that runs the whole morning and keeps the commit.

It breaks at the handoff. The file system on the volume keeps its map of blocks in one kernel’s memory, so the volume attaches to one machine at a time. And “the volume and instance must be in the same Availability Zone.” Taking it from a failed instance means a forced detach, which AWS calls a “last resort.” The instance “doesn’t get an opportunity to flush file system caches or file system metadata,” so you check the file system before you use it. (From the AWS pages on [attaching](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-attaching-volume.html) and [detaching](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-detaching-volume.html) a volume. Multi-Attach lets an `io1` or `io2` volume attach to several instances in one zone, but ext4 and XFS aren’t built for that ([Multi-Attach](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes-multi.html)).)

**A volume outlives its machine, inside one zone**

Machine A commits to an EBS volume in availability zone a, and then machine A is lost. The volume survives. Machine B, in the same zone, force-detaches the volume, attaches it, and checks the file system, and the commit is there. Machine C, in zone b, can’t attach the volume at all.

The rest of the day follows from the same fact. Three attempts mean a snapshot and three new volumes, and a volume restored from a snapshot fetches its blocks from S3 as they’re read, so it starts slow. (From AWS’s [Amazon EBS snapshots](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-snapshots.html) and [Initialize Amazon EBS volumes](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-initialize.html) pages.) And a volume bills the size you gave it, idle or not. A 20 GiB volume holding a 5 GiB workspace bills four times what it holds, and EBS can grow a volume but can’t shrink it.

## A shared file system over NFS

The fifth rung lets every machine mount the same files at once. On Amazon EFS, any number of machines mount one file system over NFS. A Regional file system stores a write across availability zones when the program calls `fsync()` or closes the file, and you pay for what you store. (From the AWS page on [EFS features](https://docs.aws.amazon.com/efs/latest/ug/features.html).) The host can be lost, and the second machine mounts the file system and finds the commit. It’s the first rung that runs the morning’s tools and gets through the middle of the day without a forced detach or a download.

It breaks on the morning’s small files. When two machines create the same name, only the server can decide which one wins. So an NFS client waits for the server on every create, every change of mode, and every close. A small file costs about four round trips. For how those calls add up, see [Anatomy of a file create](https://archil.com/guides/how-file-systems-work/anatomy-of-a-file-create). Amazon S3 Files, which serves a bucket over NFS, shows the cost on the install.

**npm ci, timed until durable**

`npm ci` of a Next.js app writes 18,602 files. On one EC2 host, timed until every write was durable, it took 10.7 s on the instance’s own EBS volume, 12.4 s on Archil, and 71.4 s on Amazon S3 Files. The animation plays the three runs side by side, 10 times faster than they ran.

The install takes almost seven times as long over NFS as on the volume. EFS itself took 86.3 s for `npm ci` of a create-next-app project in a separate benchmark, 11 times as long as Archil in the same run. (That benchmark stopped the clock when each command returned. For the hosts, versions, and commands behind both benchmarks, see [File systems on one host](https://docs.archil.com/reference/benchmarks/file-systems).) NFS clients also cache what they’ve seen, so a second machine sees a new file only when its cached copy of the directory expires. Archil’s clients do the same by default.

**Until another machine sees a new file**

We created a file on one client and measured how long a second client took to see it. Archil, when the reader runs `archil invalidate-cache` first: about 70 ms. Mountpoint for Amazon S3: 284 ms, with no metadata cache by default. A Daytona Volume: about 360 ms, the median of 10 runs. Amazon S3 Files: 30 s, the length of its NFS directory attribute cache. Archil with default client caching: 30 s, its client cache expiry. A Modal Volume: not within 60 s, because a running sandbox sees another sandbox’s commit only after it reloads the Volume.

Forks are the last cost on this rung. EFS has no clone, so each attempt copies every file over NFS and pays the round trips of the install again.

## JuiceFS, and a database you run

The sixth rung stops asking one system to do everything. JuiceFS keeps names, directories, modes, links, and the map from each file to its blocks in a metadata database that you run, such as Redis or PostgreSQL. It keeps file contents as numbered blocks in your bucket. (From the JuiceFS [architecture page](https://juicefs.com/docs/community/architecture). The Community Edition is Apache 2.0.) A rename changes one row, a write in the middle of a file uploads only the new bytes, and any machine can mount it. `juicefs clone` copies only metadata, so forks share blocks. It’s the most complete of the do-it-yourself designs.

**Two systems to keep alive**

A machine running the JuiceFS client commits. The names, modes, and map of blocks go to a metadata database you run, such as Redis or PostgreSQL, which you have to back up, fail over, and size. The file contents go to your bucket as numbered blocks. Lose the database, and the bucket holds blocks with no names.

The cost is the database, which you now operate, and the file system is only as safe as it is. JuiceFS’s own guide to Redis says the default log sync means “you may lose the writes from the last second.” It also says failover “does not guarantee that acknowledged writes are retained during failures,” and “Redis data needs to be backed up regularly.” (From JuiceFS’s [Redis best practices](https://juicefs.com/docs/community/redis_best_practices/).) Lose the database, and JuiceFS’s docs say “you cannot find the original file directly in the object storage.”

The database also sits in the path of every small file. With Redis on ElastiCache in the same availability zone, our `npm ci` took 161.6 s with JuiceFS’s defaults, 15 times as long as on the volume. Its write-back mode is faster because it stages blocks on local disk and uploads them later. Its docs warn that “if write cache data suffers loss before upload is complete, file data is lost forever.” (From JuiceFS’s guide to [cache](https://juicefs.com/docs/community/guide/cache/). Our setup is in the [file system benchmark](https://docs.archil.com/reference/benchmarks/file-systems#archil-vs-juicefs-vs-amazon-s3-files).) That is the second rung’s gap, back again.

## One day on every rung

Run the day yourself. Pick a design, and each step shows whether the workspace comes through it and how long it took where we measured it. The idle week is priced at list price for a workspace holding 5 GiB. Where we have no measurement, the step says what the vendor documents instead.

**An agent’s day**

One workspace through one day on seven designs, judged step by step: clone a repository, run `npm ci`, lose the host after `fsync()` returns, hand off to a second machine, fork three attempts, and sit idle for a week. Times are our measurements, and idle costs are for a workspace holding 5 GiB at us-east-1 list prices.

On the sandbox’s disk, the clone took 15.5 s and the install 10.7 s, measured on an EBS gp3 volume. The host loss takes the workspace, the second machine starts empty, forks are local copies, and keeping the sandbox up through the idle week costs about $28.

Uploading to S3 works at the same local speed, but the host loss takes everything since the last upload. The second machine and each fork download the last upload, and the idle week costs $0.03 in S3 Standard.

On a mounted bucket, the clone and the install fail, because Mountpoint can’t rename on a general purpose bucket or create symbolic links. The commit survives the host, the second machine mounts the bucket in 299 ms, forks are full copies, and the idle week costs $0.03.

On a network volume, the clone and the install run at the same speeds as the sandbox’s disk. The volume survives the host, but the handoff needs a forced detach, a machine in the same zone, and a file system check. Forks are a snapshot and three volumes that load lazily, and the idle week costs $0.37 for a 20 GiB volume.

On NFS, EFS took 81.5 s to clone React and 86.3 s to install a create-next-app project in a separate benchmark, where Archil took 9.8 s and 7.6 s. The commit survives, the handoff works, forks are full copies over NFS, and the idle week costs $0.35.

On JuiceFS with its defaults, the clone took 247.4 s and the install 161.6 s, or 29.2 s and 53.8 s in write-back mode. The commit is as safe as the metadata database, the handoff works, forks copy only metadata, and the idle week costs $0.03 in S3 plus the database.

On Archil, the clone took 19.8 s and the install 12.4 s. The commit survives. The lost client’s checkout doesn’t expire. Archil marks it orphaned, usually within a minute, and the second machine takes it over with `--force`, losing only writes that were never synced. It then mounts in 44 ms. Forks are one 13 ms checkpoint and three 33 ms branches at the median, and the idle week costs $0.03, billing the archive rate on every byte.

Every rung gets through part of the day. The fast ones lose the host or stay with one machine in one zone. The ones that share are slow on small files, can’t run git, or need a database you run.

## Where the authoritative copy lives

Each rung failed for its own reasons, but the reasons share a shape. Each design decides where the authoritative copy of the workspace lives, and which machine may answer a read or a write without asking anyone else. The following figure puts each rung where it keeps that copy.

**Where each rung keeps the truth**

Where each rung keeps the authoritative copy of the workspace. The sandbox’s disk keeps it on the machine: fast, and gone with the host. Uploading to S3 keeps it on the machine first and in the bucket later: fast, and it loses what wasn’t uploaded. A mounted bucket keeps it in the bucket: safe and shared, without rename or links. A network volume keeps it on a volume: safe and fast, for one machine in one zone. NFS keeps it on a server: safe and shared, with a round trip for each change. JuiceFS keeps the map in a database and the blocks in the bucket: complete, with a database to run.

A design that fixes the authority in one place gets part of the day, and climbing to the next rung only moves the authority somewhere else. The way off the ladder is to stop fixing it in one place: keep the truth remote and replicated, and lend the right to answer to whichever machine is doing the work. The idea is decades old, and [Where the truth lives](https://archil.com/guides/storage-for-agents/where-the-truth-lives) covers why it works and what building it takes.