ARCHIL

Evaluating storageChapter 5 of 7

Can you fork it?

The short answer

An agent harness forks the workspace for every attempt and every undo. If a fork is a copy, each one costs time and storage in proportion to the workspace. A copy-on-write fork is ready in milliseconds at any size and stores only what each attempt changes, as long as it includes the writes your program just made.

Hunter LeathFounder & CEO, Archil

8 min readChecked November 2026Read as Markdown

Your harness is about to try five fixes for a failing test, each in its own copy of the workspace. The workspace is 100 GiB: a repository, its dependencies, build caches, and a test dataset. If your storage makes a fork by copying, each attempt waits while 107 GB is read and written again. At 500 MB/s, a brisk rate we assume here rather than one we measured, that's about 215 seconds per attempt before the agent runs a single command.

Copy-on-write storage makes the same fork without copying the data. On Archil, a checkpoint takes 13 ms and a branch 33 ms at the median, whatever the size of the disk.The p95s are 19 ms and 36 ms, measured with our CLI from a c7gn.2xlarge in AWS us-east-1. The branch time includes opening a new TLS connection. For more information, see Limits and the latency benchmark in our docs. The rest of this chapter is about what that difference does to a harness, how to tell which kind of fork a vendor offers, and the one property that makes a fork trustworthy.

· Copy or branchPlayed in real time. The copy's progress bar is drawn to scale.

A full copy and a branch of a 100 GiB workspace start at the same moment, played in real time. After 5 seconds, the copy, at an assumed 500 MB/s, has written 2,500 MB of about 107,000 MB and has about 3.5 minutes to go. The branch was ready in 33 ms, and the agent has been working in it ever since.

What a copy costs a harness§

Harnesses fork in two patterns. The first is fan-out: take one checkpoint, branch it once per attempt, keep the attempt that passes, and throw the rest away. Each attempt sees a workspace that no other attempt can touch.

· Five attempts from one checkpointFour branches fail their tests and are thrown away. Nothing was copied.

The harness takes one checkpoint of the workspace and makes five branches from it, one per attempt. Each agent applies a different plan in its own branch and runs the tests. Attempt 4 passes, and its branch is kept. The other four fail, and their branches are thrown away. No attempt copied the workspace.

The second is undo. A file system commits work one system call at a time, but an agent's unit of work is the turn. A git clone or an npm install is thousands of file operations, and a turn that fails halfway leaves half of them behind. If the harness checkpoints after each turn that succeeds, it undoes a bad turn by branching from the last good checkpoint.

· Checkpoint every turnThe failed turn's writes stay on the old line. The retry never sees them.

An agent's harness takes a checkpoint after each turn that succeeds: turn 1 runs git clone, turn 2 runs npm install. Turn 3 edits the source and breaks the tests. Instead of repairing the damage, the harness branches from the checkpoint after turn 2 and runs turn 3 again there, where the tests pass. The failed turn's writes stay behind on the old line.

With copies, both patterns cost time and storage for every fork. Sixty-four attempts on the 100 GiB workspace means 64 × 215 seconds, about 3.8 hours of copying, and 6,400 GiB of storage that is almost all identical. Running the copies in parallel only divides the same bytes across the bandwidth you have. A harness that pays that much forks less: it runs fewer attempts and checkpoints less often.

Try your own numbers in the following calculator. The storage figures for branches follow our metering rule: data that branches share is billed once, plus what each branch adds.For the rule and a worked example, see Metering in our docs. Archil bills active storage at $0.20 per GiB-month.

50
100 GiB
5% · 5 GiB
Try
Full copies
every attempt gets the whole disk
Each tile is a whole copy of the disk.
Ready to start
3 h
50 × 100 GiB ÷ 500 MB/s
Storage billed
5,100 GiB
(1 + 50) × 100 GiB
Copy-on-write branches
every attempt shares the base
Filled: what each branch owns. Empty: read from the shared base.
Ready to start
1.7 s
1 checkpoint + 50 branches, one at a time
Storage billed
350 GiB
100 GiB + 50 × 5 GiB
Time until every attempt can start (log scale)
copies3 h
branches1.7 s
10 ms1 s1 min1 h
Storage billed
copies5,100 GiB
branches350 GiB
50 attempts on a 100 GiB disk, each changing 5%. Full copies: 3 h to make and 5,100 GiB stored. Branches: 1.7 s and 350 GiB stored.
· What a fan-out costsThe copy rate is the assumed 500 MB/s. Branch times are our measured medians, made one at a time.

The calculator compares full copies with copy-on-write branches for any number of attempts, disk size, and share of the disk each attempt changes. Copies cost the whole disk per attempt in time and in storage. Branches cost one 13 ms checkpoint plus 33 ms per branch, and store the base once plus each attempt's changes. With the example from our metering docs (50 attempts on a 100 GiB disk, each changing 5 GiB), copies take about 3 hours and store 5,100 GiB. Branches take about 1.7 seconds and store 350 GiB.

How a fork can cost nothing§

A file system already keeps a map from each file to the blocks that hold its data. A copy-on-write fork doesn't copy the blocks. It starts a second map that points at the same blocks, so the work is the same at 1 GiB as at 10 TiB. When either side later writes, the storage puts the new data in a new block and repoints only that side's map. The two sides share every block that neither has changed.

· Shared blocks, then one new oneThe parent still reads C. The branch reads C′. Both still share A, B, and D.

A disk's map points at four blocks, A to D. A checkpoint freezes the map. A branch starts with its own map pointing at the same four blocks, so no data moves. When the branch rewrites C, the storage writes a new block, C′, and repoints only the branch. The parent still reads C, and the two keep sharing A, B, and D.

Three things follow that you can use when you evaluate a vendor. A fork whose time grows with the size of the disk is a copy, whatever it's called. Storage for a fork grows with what the attempt changes, so an attempt that rewrites the whole workspace costs as much as a copy. And deleting a fork is harder than making one, because the storage has to work out which blocks no other fork still uses. Ask whether you can delete forks, and when their space comes back.

A fork is only as current as the storage§

A fork captures whatever the storage holds at that moment. Storage that acknowledges fsync() from a local drive and uploads the data later makes its forks from the uploaded copy, which lags. We tested this on Vercel Drives. A sandbox wrote a 4 KiB file and a 64 MiB file, called fsync(), and then the script forked the Drive and read the fork from a new sandbox.The script and every result are in Vercel Drive fsync durability in our docs. For why a write acknowledged from a local drive isn't safe yet, see When is a write safe?

The following table shows what the forks held, by the time between fsync() returning and the fork request.

Time after fsync() returnedForks missing the new file
0 s11 of 11
1 s8 of 8
5 s9 of 10
30 s9 of 10
60 s0 of 10

A harness that forks the moment a turn ends gets a workspace that is missing the turn. If the turn was an npm install, the fork holds part of one. Vercel's docs describe a Drive fork two ways: as "copying the drive's data", and as a drive that inherits its parent's data "but stores its own changes independently". We didn't time Drive forks, so we can't say which one you pay for. The first test in the following section finds out whether a fork's time grows with the disk.Both phrases are from Vercel's Drives documentation.

Testing a fork yourself§

Two tests tell you most of what you need. Run them on a scratch disk with the vendor's own fork command in place of fork.

Terminal
# 1. Is it a copy? Time a fork of a small disk and of your largest workspace.
time fork small-disk try-small     # 1 GiB
time fork big-disk try-big         # your largest workspace
 
# 2. Does it include what fsync() made safe? Write, sync, fork at once, compare.
dd if=/dev/urandom of=/work/probe bs=1M count=64 conv=fsync
sha256sum /work/probe
fork big-disk try-probe            # immediately
# Mount try-probe on another machine, then:
sha256sum /mnt/try-probe/probe     # must match, every time

Run the second test 10 times. One mismatch means the storage forks from a copy that lags its acknowledgments. Then delete the forks and watch whether the storage you're billed for goes down.

Forks elsewhere§

The following list covers how other kinds of storage fork.

  • Local copy-on-write file systems, such as btrfs and ZFS, snapshot and clone in moments. The fork lives on the same disk, so no other machine can mount it.
  • Amazon EBS stores incremental snapshots in Amazon S3. A volume restored from one fetches its blocks from S3 as they're read, and can run with higher latency until every block has arrived.From AWS's Amazon EBS snapshots and Initialize Amazon EBS volumes pages.
  • Amazon EFS has no native branching. You copy the data.
  • JuiceFS clones a file or directory by copying only its metadata. A directory clone isn't atomic if the source changes during it.From JuiceFS's clone documentation.
  • Tigris forks buckets without copying data. A fork is a bucket, so an agent that runs git or npm needs a file system on top of it.

What Archil does§

In AWS regions, every Archil-managed disk can checkpoint and branch, and Archil-managed disks are the default. A branch is a disk in its own right. You can mount it on any machine, including one other than its parent's, and run two branches of a disk on separate machines at once. End to end, checkpointing a disk, branching it, and mounting the branch took about 0.9 s from inside a Vercel sandbox and about 1 s from a Modal sandbox. A 1 GiB file was intact in the branch both times.

Terminal
archil checkpoints create /mnt/archil before-attempts
archil branches --region aws-us-east-1 create dsk-0123456789abcdef attempt-1 \
    --from-checkpoint before-attempts
sudo archil mount myorg/mydisk /mnt/attempt-1 \
    --region aws-us-east-1 --branch attempt-1

Before you take a checkpoint, call fsync() on the files the turn wrote. Archil's fsync() returns once the write is stored redundantly across Availability Zones.

Our sandboxes fork the same way. sandbox.fork() pauses a running sandbox, checkpoints its root disk, branches it, and resumes the original. The fork starts from the original's exact state.

· Forking a running sandboxThe original pauses for the checkpoint and branch, then resumes.

When you call sandbox.fork() on a running Archil sandbox, the sandbox pauses. Its root disk is checkpointed in about 13 ms and branched in about 33 ms, and the original resumes. The fork starts from the original's exact state, on its own branch.

Archil's forks fall short in four ways today:

  • Archil-managed disks only. A disk that syncs to your own bucket can't checkpoint or branch.
  • No deleting. You can't delete an individual branch or checkpoint, so the data an abandoned attempt wrote stays on the disk. If you fork thousands of times a day, plan for that until we ship deletion.
  • No merging. Branches diverge and don't merge back into their parent. To keep an attempt, you carry on from its branch.
  • Not a security boundary. Credentials cover a whole disk, every branch and checkpoint included. Give each untrusted tenant its own disk.

A sandbox with other disks mounted must be stopped before it forks, and the fork shares those disks rather than branching them. For how copy-on-write works underneath, block by block, and why deleting a fork is the hard part, see Copy-on-write, block by block.