# Evaluating storage for agents Source: https://archil.com/guides/evaluating-storage The seven questions to ask before you pick storage for agents and sandboxes, how to test any vendor’s answer, and how each kind of storage answers them. ## The seven questions 1. **Is it really a file system?** Git, npm, SQLite, and most agent tools assume that renames are atomic, files can be edited in place, and directories can be listed. Storage that only resembles a file system breaks them in ways that show up late. - What you’ll notice if you get it wrong: Tools fail late and in odd ways: a rename that isn’t atomic, a symlink that can’t be created, a database that can’t write in place. - How to check: Run a script that renames over an existing file, overwrites bytes in the middle of a file, creates a symlink and a hard link, and sets the executable bit. - Read more: [Is it really a file system?](https://archil.com/guides/evaluating-storage/is-it-really-a-file-system.md) 2. **When is a write safe?** Sandboxes are moved and retired without notice. A write is safe only if it was replicated to other machines before fsync returned. - What you’ll notice if you get it wrong: After a host is lost, files that your program had already synced are missing or out of date. - How to check: Time 4 KiB writes with `fdatasync` using fio. A median well under a millisecond means the acknowledgment came from the machine you’re on. - Read more: [When is a write safe?](https://archil.com/guides/evaluating-storage/when-is-a-write-safe.md) 3. **How fast is it on small files?** Clones, installs, and SQLite commits are thousands of small files. Their speed depends on round trips per file, not on throughput. - What you’ll notice if you get it wrong: `git clone` and `npm install` take a minute or more on storage that benchmarks well on throughput. - How to check: Time `npm ci` on a real project, and stop the clock when the writes are durable, not when the command returns. - Read more: [How fast is it on small files?](https://archil.com/guides/evaluating-storage/how-fast-is-it-on-small-files.md) 4. **Can many machines use it at once?** Parallel agents, a reviewer beside a writer, and a sandbox that picks up where another stopped all need the same files on more than one machine, with rules for who writes. - What you’ll notice if you get it wrong: A second agent reads a stale file, or two agents overwrite each other’s changes without an error. - How to check: Write a file on one machine and read it on another. Measure how long the change takes to appear, and what happens when both write. - Read more: [Can many machines use it at once?](https://archil.com/guides/evaluating-storage/can-many-machines-use-it-at-once.md) 5. **Can you fork it?** Agents try things. A copy-on-write fork gives every attempt its own workspace in milliseconds, and you throw the bad ones away. - What you’ll notice if you get it wrong: Each new attempt waits minutes for a copy, so the harness runs fewer attempts and checkpoints less often. - How to check: Fork a 1 GiB workspace and your largest one. If the times differ, the fork is a copy. - Read more: [Can you fork it?](https://archil.com/guides/evaluating-storage/can-you-fork-it.md) 6. **What does it cost when nothing is running?** Most agent workspaces sit idle most of the time. Storage that’s billed by provisioned size, or that’s tied to a running machine, charges you for the waiting. - What you’ll notice if you get it wrong: The storage bill stays flat when the agents stop working, or the files disappear along with the sandbox. - How to check: Price one workspace for a month in which an agent uses it for an hour a day. - Read more: [What does it cost when nothing is running?](https://archil.com/guides/evaluating-storage/what-does-it-cost-when-nothing-is-running.md) 7. **Where does your data live?** If the files end up as ordinary objects in your own bucket, every other tool can read them and leaving is a copy, not a migration. - What you’ll notice if you get it wrong: Leaving the vendor means a migration, and other tools can read the files only through the vendor. - How to check: Write a file, then look in the bucket: is it there, at its path, as an ordinary object? - Read more: [Where does your data live?](https://archil.com/guides/evaluating-storage/where-does-your-data-live.md) ## The scorecard The scorecard rates architectures, not products, because products built the same way share their strengths and limits. For more information about each product, see [the product comparisons](https://archil.com/compare). | | Local disk | Local disk, uploaded later | Network block volume | NFS | Metadata database | Object storage over FUSE | Archil | | --- | --- | --- | --- | --- | --- | --- | --- | | A real file system | Yes: all of POSIX, on one machine | Yes: all of POSIX on the machine; one writer | Yes: all of POSIX; one machine at a time | Yes: all of POSIX; close-to-open between machines | Yes: a POSIX file system, by its own description | No in-place edits; renames copied or refused | Yes: all but ACLs and some rename flags | | Writes survive a lost machine | No: survives a reboot, not a lost host | No: lost until the upload finishes | Yes: replicated within one zone before acknowledging | Yes: stored by the server first; One Zone is single-zone | Partly: kept by default; write-back mode uploads later | Yes: kept once fsync uploads the object | Yes: replicated across zones before fsync returns | | Fast on small files | Yes: fastest | Yes: local speed, before the upload | Yes: close to local | No: a round trip per call | No: a database round trip per call | No: an object request per call | Yes: within seconds of local disk on installs; slower when clients contend for one path | | Many machines at once | No: one machine | No: one writer; others get stale snapshots | No: one machine at a time; hand off by detaching | Partly: many writers, close-to-open; the last write wins silently | Partly: close-to-open; the last write to each byte stays | Partly: fast visibility; no in-place edits or locks | Partly: many writers, at remote speed or at local speed on checked-out paths; new files appear elsewhere in about 30 s unless the reader refreshes | | Instant forks | Partly: instant snapshots with btrfs, on that machine only | Partly: vendor snapshots and forks, which can miss recent writes | Partly: snapshots restore as a new volume, loaded lazily | No native branching on EFS | Partly: juicefs clone per file or directory, not atomic | Partly: bucket forks on Tigris; none on S3 | Yes: 13 ms checkpoints, 33 ms branches, on managed disks; branches can’t be deleted yet | | Cheap while idle | No: kept only while you pay for the machine | Partly: a stored copy or the running machine, by vendor | No: full provisioned size, every hour; can’t shrink | Partly: bytes stored, at file-storage rates | Partly: object-storage rates, plus an always-on database | Yes: object-storage rates; nothing has to run | Yes: active rate only near use; archive or bucket rate otherwise | | Data in your bucket | No: on one machine | No: the vendor’s storage, in its account | No: a disk image; snapshots S3 tools can’t read | Partly: readable only over a mount on EFS; S3 Files writes ordinary objects after 60 s without writes | Partly: your bucket, but as numbered blocks readable only through JuiceFS | Yes: ordinary objects, for what it can write | Partly: managed disks, the default, use our format, read through our S3 API or a mount; bucket-synced disks leave ordinary objects, without modes or links | The following list shows the products built each way: - **Local disk:** A sandbox’s own disk, EC2 instance store - **Local disk, uploaded later:** Vercel Drives, Fly.io Sprites - **Network block volume:** Amazon EBS, Google Persistent Disk - **NFS:** Amazon EFS, Amazon S3 Files, Google Filestore - **Metadata database:** JuiceFS - **Object storage over FUSE:** Mountpoint for Amazon S3, s3fs, Cloud Storage FUSE - **Archil:** Archil disks ## Questions to ask any vendor - Is `rename()` atomic over an existing file? Can a program overwrite bytes in the middle of a file without rewriting all of it? - Which POSIX features don’t you support: hard links, symbolic links, permissions, or extended attributes? - When `fsync()` returns, how many machines hold the data, and are they different from the one my code runs on? - If the host dies a millisecond after `fsync()` returns, what does the next reader see? - How many network round trips does it take to create, write, and close one small file? - When one machine writes a file, how long until another machine reads the new contents? What happens when both write? - How long does a fork take at 1 GiB, and at the size of my largest workspace? - Am I billed for each fork’s full size, or only for what it changes? - What do I pay for a workspace that nobody has touched for a week? - Where are my files stored, in what format, and how do I get all of them out? ## 1. Is it really a file system? Source: https://archil.com/guides/evaluating-storage/is-it-really-a-file-system · Part of Evaluating storage for agents · Facts checked 2026-11-04 **The short answer:** Git, npm, pnpm, and SQLite depend on atomic `rename()`, writes at byte offsets, symbolic and hard links, and permission bits. Object storage over FUSE refuses those calls or fakes them with whole-object copies, whereas NFS and block volumes are real file systems. Archil supports all of them, but not POSIX ACLs or some `renameat2` flags. Run an agent’s usual tools on storage that only looks like a file system, and the first error usually comes from a tool, not from the storage. On Mountpoint for Amazon S3, `git init` can’t finish: git writes `.git/config` to a lock file and renames it into place, and Mountpoint refuses renames on general purpose buckets. On storage without symbolic links, `npm install` fails. A script that was executable in the repository checks out without its executable bit, and `./build.sh` answers `Permission denied`. Other failures make no noise. SQLite changes one 4 KiB page, and an s3fs mount uploads the whole database file again. pnpm, which normally hard-links each package from a shared store, finds no hard links and copies every file instead. Nothing reports an error, and every loop the agent runs gets slower. These tools were written against POSIX, the interface that Unix file systems share, and agents run them because they were trained on them. This page covers the five calls behind those failures, a script that tests for them, and where Archil falls short. For the model underneath, from inodes to how an object store maps onto them, see [From a path to an inode](https://archil.com/guides/how-file-systems-work/from-a-path-to-an-inode). **Five calls, five failures** Five commands and the call each one depends on. `git commit` renames `index.lock` over `index`, and without an atomic rename git can’t update its index. SQLite writes a 4 KiB page at an offset in the database file, and without in-place writes each commit rewrites the whole file. `npm install` creates symbolic links in `node_modules/.bin`, and without symlinks the install fails. `pnpm install` hard-links files from its store into `node_modules`, and without hard links every file is copied. `git checkout` sets the executable bit on `build.sh`, and without mode bits `./build.sh` fails with `Permission denied`. ### What each tool calls #### git: a lock file and a rename Git changes its own files the same way every time. To update the index, a branch, or `.git/config`, it creates `.lock` with `O_CREAT|O_EXCL`, writes the new contents into it, and renames the lock over the original. The exclusive create keeps a second git process out, and the rename publishes the change. Git’s source says that readers “are guaranteed to see either the old contents of the file or the new contents”, provided the file system implements `rename(2)` atomically. (From git’s [lockfile.h](https://github.com/git/git/blob/master/lockfile.h). Writes to `.git/config` go through the same lock file, in [config.c](https://github.com/git/git/blob/master/config.c).) Storage without `rename()` stops git outright, as in the Mountpoint case at the top of this page. Mountpoint’s README says not to work on a Git repository at all. (From the [Mountpoint README](https://github.com/awslabs/mountpoint-s3) and its [semantics document](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md). Mountpoint can rename single files on S3 Express One Zone.) s3fs fakes a rename with a server-side copy and a delete, so git runs, but its README lists atomic renames among the things it can’t provide. (From the [s3fs README](https://github.com/s3fs-fuse/s3fs-fuse).) **Saving by rename** Git moves a branch by creating `main.lock`, writing the new commit id `9c2e` into it, and renaming it over `main`. The rename changes one directory entry in one step: `main` now points to inode 58 instead of inode 41, and inode 41, with no name left, is freed. A reader running `cat main` the whole time sees `a1f3` and then `9c2e`, never a missing or half-written file. #### SQLite: writes in place In its default rollback-journal mode, SQLite copies the pages it’s about to change into a journal file, then writes the changes into the database file at those pages’ offsets. (See SQLite’s [Atomic Commit](https://www.sqlite.org/atomiccommit.html), sections 3.5 and 3.9.) A transaction that changes one row writes a few KiB into a file that may hold gigabytes. An object can’t be changed in place. Mountpoint accepts writes only in order from the start of a file, so SQLite can’t update an existing database on it at all. s3fs accepts the write and rewrites the whole object when it flushes, and SQLite flushes on every commit. For a 1 GiB database, that’s 262,144 times the data that changed (1 GiB ÷ 4 KiB), on every commit. **What does this operation cost?** For five common operations, the figure compares the POSIX calls that a file system makes with what an S3 general purpose bucket must do instead. Renaming a directory of 10,000 files, 200 MB in all, is one `rename()` on a file system and 10,020 requests that copy all 200 MB on S3. Changing 4 KiB of a 1 GiB file writes 4 KiB on a file system and a new 1 GiB object on S3. Appending 100 bytes to a 64 MiB log rewrites the whole 64 MiB object. Saving by rename takes three calls on a file system and three requests on S3, but S3 writes the data twice. A hard link is one call and no data on a file system. S3 has no links, and the nearest thing is a full copy that stops tracking the original. #### npm: symbolic links in `.bin` npm installs each package’s commands into `node_modules/.bin` as symbolic links, and that’s how `npm test` finds `jest` and `npm run build` finds `tsc`. (See npm’s [folders documentation](https://docs.npmjs.com/cli/configuring-npm/folders). The links are made in [bin-links](https://github.com/npm/bin-links/blob/main/lib/link-gently.js), which passes a failed `symlink()` up as an error.) On storage that refuses `symlink()`, the install fails. npm’s `bin-links` setting can turn the links off, which its documentation suggests because “some file systems don’t support symlinks”. Then `.bin` stays empty, and scripts can’t find the commands they call. Git meets the same gap more quietly. On storage without symbolic links, a clone checks each link out as a small plain file that contains the link’s target, and the build that follows the link is what fails. (This is git’s `core.symlinks` setting. See [git-config](https://git-scm.com/docs/git-config).) #### pnpm: hard links from a store pnpm keeps one copy of each package file in a content-addressable store and hard-links it into every project’s `node_modules`, so a hundred projects that use a package share one copy on disk. Its default `node_modules` layout is built from symbolic links as well. (See pnpm’s [motivation](https://pnpm.io/motivation) and the [`packageImportMethod`](https://pnpm.io/settings#packageimportmethod) setting.) When it can’t hard-link or clone, pnpm falls back to copying. The install succeeds, writes every byte of every package, and takes that much space again in each project. #### The executable bit Git records one permission bit per file: whether it’s executable. When `git init` or `git clone` starts, it flips that bit on `.git/config` and reads it back to see whether the file system keeps it. If the bit doesn’t stick, git sets `core.fileMode` to false and stops tracking it. (The probe is in git’s [setup.c](https://github.com/git/git/blob/master/setup.c). Git’s index stores only mode 0755 or 0644 for a regular file.) Mountpoint, for example, assigns modes at mount time and `chmod` can’t change them. On storage like that, `git status` stays clean, and the script that should run doesn’t. ### Lookalikes and real file systems Two kinds of network storage pass all five tests, because both run a real file system. A network block volume, such as EBS or Persistent Disk, runs ext4 or XFS in your own kernel. NFS, as in Amazon EFS or Filestore, sends each call to a server whose file system carries it out, and its protocol says that a `RENAME` “must be atomic to the client”. (From [RFC 1813](https://www.rfc-editor.org/rfc/rfc1813), section 3.3.14.) Their limits lie elsewhere: a block volume serves one machine at a time, and NFS waits a round trip for each create and rename. For more information, see [How fast is it on small files?](https://archil.com/guides/evaluating-storage/how-fast-is-it-on-small-files) and [Can many machines use it at once?](https://archil.com/guides/evaluating-storage/can-many-machines-use-it-at-once) Object storage over FUSE fails because the bucket underneath has no rename, no partial writes, and no links. A FUSE client can refuse those calls, as Mountpoint does, or fake them with copies, as s3fs does. Neither can add an operation that the store doesn’t have. Whole-file APIs fail more completely. Some agent toolkits give the model `readFile` and `writeFile` calls over a database or a bucket. Git, npm, and SQLite call the kernel, not the toolkit, so they don’t run there at all. Only the commands that the toolkit reimplements do. **Where the file system runs** The figure sends the same `rename()` to three architectures. On a network block volume, ext4 or XFS in your own kernel carries it out and sends a few block writes, so you get all of POSIX on one machine. Over NFS, the client sends one `RENAME` and the server’s file system carries it out, so you get all of POSIX on many machines. Over a FUSE client on S3, the rename becomes a copy and a delete for every object, because the bucket has no rename to call. ### Check it yourself The following script tests each call in a few seconds. Run it on the storage you’re evaluating, with the path of a directory there: ```sh #!/bin/sh ## Usage: sh posix-check.sh /path/on/the/storage/you/are/testing dir=${1:?usage: sh posix-check.sh DIR}/posix-check.$$ mkdir "$dir" && cd "$dir" || exit 1 check() { if eval "$2"; then echo "ok $1"; else echo "FAIL $1"; fi; } ## Rename over an existing file 200 times while a reader watches for a gap. echo v0 > target ( i=1; while [ $i -le 200 ]; do echo "v$i" > tmp && mv -f tmp target || exit 1; i=$((i+1)); done ) & gaps=0 while kill -0 $! 2>/dev/null; do [ -s target ] || gaps=$((gaps+1)); done check "rename over an existing file" 'wait $! && [ $gaps -eq 0 ]' mkdir -p a/b && echo x > a/b/f check "rename a directory" 'mv a z && [ -f z/b/f ] && [ ! -e a ]' ## Overwrite 6 bytes in the middle of a 1 MiB file. dd if=/dev/zero of=big bs=1024 count=1024 2>/dev/null printf 'abcdef' | dd of=big bs=1 seek=524288 conv=notrunc 2>/dev/null check "write at an offset" '[ $(wc -c < big) -eq 1048576 ] && [ "$(dd if=big bs=1 skip=524288 count=6 2>/dev/null)" = abcdef ]' check "symbolic link" 'ln -s big sym && [ "$(readlink sym)" = big ] && [ -f sym ]' ## A hard link is a second name for the same file, not a copy. ln big twin 2>/dev/null; printf 'Z' | dd of=twin bs=1 conv=notrunc 2>/dev/null check "hard link" '[ "$(ls -l big | awk "{print \$2}")" = 2 ] && [ "$(dd if=big bs=1 count=1 2>/dev/null)" = Z ]' printf '#!/bin/sh\necho ran\n' > run.sh check "executable bit" 'chmod +x run.sh && [ "$(./run.sh 2>/dev/null)" = ran ]' check "fsync" 'python3 -c "import os,sys; fd=os.open(sys.argv[1], os.O_WRONLY); os.write(fd, b\"y\"); os.fsync(fd)" big' cd .. && rm -rf "$dir" ``` Every line should say `ok`. The script has three blind spots. It runs on one machine, so it can’t show what a second client sees during a rename. Run the reader loop on another machine for that. It checks that `fsync()` returns, not that the data survives a lost host. For that test, see [When is a write safe?](https://archil.com/guides/evaluating-storage/when-is-a-write-safe) And it checks behavior, not cost: s3fs passes the offset write while it uploads the whole file. For a broader test, [pjdfstest](https://github.com/pjd/pjdfstest) is a public suite that exercises POSIX system calls one by one, including `chmod`, `link`, `rename`, `symlink`, and `unlink`. It runs as root from a directory on the file system under test. ### Where Archil falls short Archil is a POSIX file system that mounts on Linux through FUSE. It supports random writes and appends, atomic renames of files and whole directories, symbolic and hard links, modes and ownership, and files that stay open after they’re unlinked. Git, SQLite, PostgreSQL, and npm run on it unchanged. (The full list is in [Application compatibility](https://docs.archil.com/details/support).) It has two gaps: - **No POSIX ACLs.** Mode bits and ownership work. Access control lists don’t. - **Some rename flags.** A `renameat2` call with `RENAME_NOREPLACE`, `RENAME_EXCHANGE`, or `RENAME_WHITEOUT` fails with `EINVAL`, so an Archil disk can’t be an overlayfs upper directory. (A persistent sandbox’s root volume is ext4 and supports these flags, overlayfs, and Docker’s `overlay2`. The gaps apply to disks mounted through FUSE.) A disk synced to your own bucket adds a translation that mounted clients don’t see. The clients get real links, but the bucket stores a symbolic link as a small text file that holds its target, and a hard link as a full copy. A pnpm store on such a disk is shared on the mount and copied in full in the bucket. (See [Data sources](https://docs.archil.com/concepts/data-sources). For more information about what the bucket holds, see [Where does your data live?](https://archil.com/guides/evaluating-storage/where-does-your-data-live)) ## 2. When is a write safe? Source: https://archil.com/guides/evaluating-storage/when-is-a-write-safe · Part of Evaluating storage for agents · Facts checked 2026-11-04 **The short answer:** A write is safe once `fsync()` has returned and the data is stored on more than one machine. Some persistent disks for agents acknowledge `fsync()` from a drive inside the machine and upload the data later, so a write they called safe is lost if that machine is lost. Ask any vendor where `fsync()` returns from, and time it yourself. We ran a test on Vercel Drives, the persistent disks that Vercel offers its sandboxes. A sandbox wrote a file and called `fsync()`, the system call a program makes to ask that its writes be stored permanently. As soon as `fsync()` returned, we took a snapshot of the Drive and opened it in a second sandbox. The file wasn’t there in any of 21 trials. Forks of the Drive taken at the same moment were missing it too. **A Vercel Drive after fsync returns** A sandbox wrote a 4 KiB file and a 64 MiB file, called `fsync()` on each, and synced the directory. After a delay, a new sandbox read the Drive through a snapshot or a fork. Taken the moment `fsync()` returned, 10 of 10 snapshots, 11 of 11 snapshots taken while the writer kept writing, and 11 of 11 forks were missing the file. At 1 second, every snapshot and fork still lacked it. At 30 seconds, 7 of 10 snapshots and 9 of 10 forks lacked it, and 4 of 10 snapshots taken while the writer kept writing held a 64 MiB file whose last 12 to 28 MiB read as zeros. From 60 seconds on, the file was intact in every trial but one. An Archil disk mounted in a new sandbox the moment `fsync()` returned had the file in 20 of 20 trials. A snapshot isn’t a crash. But a snapshot is read from the storage that the Drive would be rebuilt from if its host failed, and for most of the first minute that storage didn’t have the write. Thirty seconds after `fsync()` returned, some snapshots held the file with its last megabytes reading as zeros. Only from about a minute on did snapshots and forks reliably hold it. (The script and every result are in [Vercel Drive fsync durability](https://docs.archil.com/reference/benchmarks/vercel-drive-durability). Vercel describes Drives as having “NVMe speed writes” in its [Drives documentation](https://vercel.com/docs/sandbox/concepts/drives).) If the host had died during that minute, the write would have died with it, after the application had already been told it was safe. That matters most for the software that relies on `fsync()` most. When a database tells you a transaction committed, it has called `fsync()` and waited for it to return. A user is told their account exists, or a traveler that their seat is booked. Lose the write after that, and the account is gone when they log in and the seat goes to someone else. ### Machines that don’t come back The Linux manual page for `fsync()` promises that once it returns, “all changed information can be retrieved even if the system crashes or is rebooted.” The guarantee was written for a machine with a disk inside it, where the worst that usually happens is a restart. In the cloud, the machine itself is often what fails. When AWS detects an “irreparable failure of the underlying host” for an EC2 instance, it schedules the instance to stop or terminate, and data on the host’s own drives doesn’t survive a stop or a termination. (From the AWS pages on [instances scheduled to retire](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/schedevents_actions_retire.html) and [instance store data persistence](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-store-lifetime.html). The same table says instance store data survives a reboot and a power failure, which is the case `fsync()` was designed for.) A sandbox runs on whichever host has room, so its local disk is that host’s disk, and it goes wherever that host goes. Try it in the following figure. Pick what fails, then drag the moment of the crash. **Pull the plug** A program writes `v1`, writes `v2`, calls `fsync()`, and then writes `v3`. Each band shows what the program is guaranteed to read back if the crash happens at that moment, for three kinds of failure. After `fsync()` returns, every band should show `v2` or later. When only the program crashes, every kind of storage keeps every write, because the operating system still holds them. When the power is cut, every kind keeps the writes that `fsync()` covered, because the machine restarts with its disks. When the machine is lost, only storage that replicated the data to other machines before `fsync()` returned still has `v2`. The local disk loses everything, and the disk that uploads later loses `v2` until the upload finishes. A test that kills your program and then checks the file proves nothing about storage, because the operating system still has the write. A test that cuts the power passes on almost anything. The failure that separates storage products is the one you can’t easily stage: the machine never comes back. For how a write travels from `write()` down to the drive and out to other machines, see [Where fsync returns from](https://archil.com/guides/how-file-systems-work/where-fsync-returns-from). ### Two kinds of persistent disk Many sandbox platforms now market a persistent disk for agents, and they don’t all persist data the same way. There are two models, and the difference lies entirely in what happens when your program calls `fsync()`. **Two ways to acknowledge fsync** In the first model, the local drive acknowledges `fsync()` and an upload follows. The machine is lost before the upload finishes, and the write that the application was told was safe never arrives. In the second model, the write is replicated across availability zones before `fsync()` returns. Losing the machine loses nothing, and the data reaches the bucket later. The first model acknowledges `fsync()` once the data reaches the drive inside the machine and copies it to object storage in the background. Vercel Drives behave this way in our test. Fly.io’s documentation says a Sprite’s data “is written to fast NVMe storage” while it runs and is “backed up to durable object storage” when it goes idle. (From the [Sprites overview](https://docs.fly.io/sprites/).) Both companies are making a reasonable trade for speed. Benchmarks get faster, and a database’s write speed is its `fsync()` speed. But until the copy finishes, the write exists on one drive in one machine. The second model replicates the data to other machines before it acknowledges, so no single machine can take the write with it. EBS replicates every volume within its availability zone, and S3 stores objects across at least three zones. Local NVMe can be safe too, with a replication layer above it: PlanetScale Metal runs databases on local drives and makes sure “every write has reached stable storage in two availability zones before it’s acknowledged to the client.” (From the AWS pages on [EBS features](https://docs.aws.amazon.com/ebs/latest/userguide/EBSFeatures.html) and [S3 data protection](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html), and the FAQ on [PlanetScale Metal](https://planetscale.com/metal).) Archil uses the second model. Our client keeps your writes in memory, like a local file system, and when you call `fsync()` it sends them to our servers, which replicate them across availability zones before they acknowledge. Archil is designed for 99.999% durability for recent writes before they reach S3. Your bucket receives them later, usually tens of seconds to minutes afterward. That safety costs time on every `fsync()`. A synchronous 4 KiB write on Archil takes about 2 ms at the median. A program that waits for each commit before starting the next can make at most about 500 commits per second at that speed (1 second divided by 2 ms), which is fine for an agent’s workspace and slow for a write-heavy database. (Archil’s figures, with their method, are on the [performance page](https://docs.archil.com/details/performance).) Writes your program hasn’t synced are still only in our client’s memory, and a power cut loses them, as it would on a local disk. Unmount with `archil unmount`, which flushes, rather than `umount`, which doesn’t. ### A timing test you can run You can’t see where a write goes, but you can time it. Storing data on a second machine takes at least one network round trip and a write on the far side. A drive inside the machine needs neither. Run this on the storage you’re evaluating: ```bash ## About 1,000 small writes, each followed by fdatasync(). fio --name=sync --directory=/mnt/test --rw=write --bs=4k --size=4m \ --fdatasync=1 --ioengine=sync ``` fio prints percentiles for the `sync` latency. Compare them with our own measurements of a similar operation in the following figure. **Where fsync answers from, timed** The median time to create a file, write 4 KiB, call `fsync()`, and close it, on a log scale. Two acknowledge from a drive in the same machine: a Vercel Drive took 0.18 ms and a Fly.io Sprite’s disk took 3.5 ms. Two store the write on other machines first: Archil took 4.2 ms from an EC2 instance in the same region, and Mountpoint for Amazon S3, which uploads the object before `fsync()` returns, took 244 ms. The test only works in one direction. A median of a fraction of a millisecond, like the Drive’s, most likely came from a drive in the machine, although replicated block storage can come close: AWS designs io2 Block Express volumes, which replicate within their zone, for average latency under 500 microseconds. (From the AWS page on [EBS volume types](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volume-types.html).) A slower answer proves less. The Sprite’s disk takes almost as long as Archil and still acknowledges locally. So use the timing to decide what to ask, and ask in words a vendor can’t sidestep. When `fsync()` returns, which machines hold the data, and is the one running my code among them? If that machine dies a millisecond later, what does the next reader see? ## 3. How fast is it on small files? Source: https://archil.com/guides/evaluating-storage/how-fast-is-it-on-small-files · Part of Evaluating storage for agents · Facts checked 2026-11-04 **The short answer:** Agent work writes thousands of small files, so its wall-clock time is set by how many network round trips each file costs, not by throughput. Storage that answers file operations on the machine finishes an install within seconds of a local disk. Storage that asks a server about every file takes several times as long. Installing the dependencies of one Next.js app with `npm ci` writes 18,602 files. We ran that install on one EC2 instance with several file systems mounted side by side, and stopped the clock only when every write was durable. On the instance’s own EBS volume, it took 10.7 s. On Archil, it took 12.4 s. On Amazon S3 Files, which serves an S3 bucket over NFS, the same install took 71.4 s. The host, the package cache, and the bytes written were the same each time. What differed was how often each file had to wait for a server. An agent that clones a repository, installs its dependencies, and runs its tests pays that difference at every step, and wall-clock time is what you and your users notice. **npm ci, timed until durable** `npm ci` of a Next.js app writes 18,602 files. On one EC2 host, timed until every write was durable, it took 10.7 s on the instance’s own EBS volume, 12.4 s on Archil, and 71.4 s on Amazon S3 Files. The animation plays the three runs side by side, 10 times faster than they ran. ### Two jobs, five file systems The pattern holds for `git clone` and for other designs. On the same host, a clone of the Django repository took 15.5 s on local disk, 19.8 s on Archil, and 107.5 s on S3 Files. JuiceFS, which keeps its metadata in a database (here Redis on ElastiCache, in the same Availability Zone), took 161.6 s for the install and 247.4 s for the clone with its defaults. Its write-back mode stages writes on local disk and uploads them later. Timed until the upload finished, it took 53.8 s and 29.2 s. **Two jobs, timed until durable** Two developer jobs on one AWS c7gn.2xlarge in us-east-1, each timed until its writes were durable, medians of 3 runs. `npm ci` of a Next.js app (18,602 files): local EBS gp3 10.7 s, Archil 12.4 s, JuiceFS in write-back mode 53.8 s, Amazon S3 Files 71.4 s, JuiceFS with defaults 161.6 s. `git clone` of Django: local EBS gp3 15.5 s, Archil 19.8 s, JuiceFS in write-back mode 29.2 s, Amazon S3 Files 107.5 s, JuiceFS with defaults 247.4 s. A separate benchmark against Amazon EFS shows the same shape. Cloning React (7,260 files) took 9.8 s on Archil and 81.5 s on EFS. `npm ci` of a create-next-app project took 7.6 s on Archil and 86.3 s on EFS. That benchmark stopped the clock when each command returned, rather than when its writes were durable. (For the hosts, versions, and commands behind both benchmarks, see [File systems on one host](https://docs.archil.com/reference/benchmarks/file-systems).) ### Where the time goes Divide each install by its 18,602 files. On local disk, 10.7 s is 0.58 ms per file. That’s npm’s own work: reading its cache, unpacking, and making system calls that the kernel answers from memory. On S3 Files, 71.4 s is 3.84 ms per file, so each file spent about 3.3 ms waiting on the network. On Archil, each file spent about 0.09 ms. A round trip to a server in the same region is short. Our published 4 KiB read that misses the client’s cache is one round trip, and it takes 0.50 ms at the median. So each file on S3 Files waited for the equivalent of six or seven round trips. The waits come from how a network file system creates a file. A program makes four calls: `open` with `O_CREAT`, `write`, `fchmod`, and `close`. A local file system answers all four in memory. An NFS client has to send three of them to the server, as `CREATE`, `SETATTR`, and `WRITE` followed by `COMMIT`, and wait for each answer. **One file, three ways** The same four calls (open, write, chmod, close) on three kinds of storage. A local disk answers all four in memory. An NFS client waits for the server on three of them, for four round trips: CREATE, SETATTR, and WRITE plus COMMIT on close. A client that has checked out the directory answers all four in memory and sends the change later in one message with many other files. The client can’t skip the question, even when no one else is using the directory. If two machines both create `hello.txt`, only the server sees both requests, so only the server can decide which one gets the name. The client has no way to know that nobody else is looking, so it asks every time. A client can answer from memory only once it holds the right to decide: the server has handed it the directory, and nobody else can change that directory until the client hands it back. We call that a _checkout_. NFS and our docs call it a _delegation_. By default, an Archil client checks out the whole disk when it mounts it. It answers `open`, `write`, `chmod`, and `close` from memory, and it sends the changes to our servers later, many files to a message, and whenever the program calls `fsync()`. For how this idea grew out of AFS callbacks, leases, and NFSv4 delegations, see [Anatomy of a file create over a network](https://archil.com/guides/how-file-systems-work/anatomy-of-a-file-create). ### A test you can run Run the job you care about on the same host, once on local disk and once on each storage you’re considering. Stop the clock when the writes are durable, not when the command returns. A design that stages writes on a local disk and uploads them later looks like local disk until you wait for the upload. ```bash ## Set MNT to each mount in turn, local disk first. time (git clone -q file:///data/repo.git "$MNT/repo" && sync -f "$MNT/repo") time (cd "$MNT/app" && npm ci --prefer-offline --ignore-scripts && sync -f .) ``` `sync -f` flushes the whole file system that holds the path. On Archil, run `archil sync ` instead. Keep the inputs on local disk, as in the preceding commands, so the network to GitHub or the npm registry isn’t part of the time. Then divide by the number of files. The following calculator starts from our install and takes your numbers too. **From a timing to a cost per file** The calculator divides a job’s time by the files it writes. For our `npm ci` of 18,602 files, local disk is 0.58 ms per file. Amazon S3 Files adds 3.26 ms of waiting per file, about 6.5 round trips at 0.5 ms each. JuiceFS with its defaults adds 8.11 ms, about 16 round trips. Archil adds 0.09 ms, about a fifth of one round trip. ### Where Archil is slower Small files aren’t the only workload, and on several others we lose. In the same EFS benchmark, EFS was faster on the following three tests. | Test | Archil | Amazon EFS | | -------------------------------------------- | -------- | ---------- | | Sequential write, 4 GiB, then `fsync()` | 194 MB/s | 590 MB/s | | 4 KiB random writes for 30 s, then `fsync()` | 953 IOPS | 2,976 IOPS | | SQLite, one insert of 1 million rows | 11.0 s | 1.7 s | We have more work to do on throughput. A checkout makes one client fast by making the others ask first. In the default exclusive mode, other machines can mount the disk read-only, but they can’t write until the holder unmounts. To share writes, clients check out separate paths, or mount in conditional mode, where creates and renames run on the server as they do on NFS, at the speed of a remote file system. When many clients keep writing the same files, handing ownership back and forth can cost more than NFS’s round trips. Other clients see changes within about 30 seconds by default, or at once after `archil invalidate-cache`. For more information about sharing, see [Can many machines use it at once?](https://archil.com/guides/evaluating-storage/can-many-machines-use-it-at-once) Distance still counts for reads. A checkout removes the waits on files a client writes, but a file the client has never read has to cross the network. A read that also misses our servers’ cache goes to the object store, typically 10 to 30 ms on S3. From a Modal sandbox about 40 ms from our servers, Archil read a 1 GiB file into a new sandbox at 331 MB/s, and a Modal Volume read it at 624 MB/s. Keep compute in the same region as the disk, warm the cache with [`archil prefetch`](https://docs.archil.com/reference/archil-cli#prefetch) before a job reads a tree it hasn’t seen, or run the command next to the data with [serverless execution](https://docs.archil.com/compute/serverless-execution). ## 4. Can many machines use it at once? Source: https://archil.com/guides/evaluating-storage/can-many-machines-use-it-at-once · Part of Evaluating storage for agents · Facts checked 2026-11-04 **The short answer:** Almost any storage mounts on two machines. What differs is what the second machine sees, how soon, and what happens when both save the same file, which on most shared storage is that one save silently replaces the other. On Archil, many clients can write the same disk and the same files, and other clients see a change when their cached copy expires or when they ask for it. Two agents share a repository. Each one reads `plan.md`, changes a different line, and saves the file. Both saves succeed, and one agent’s change is gone. A reviewer agent on a third machine runs the tests right after the writer commits, and it tests the code from before the commit. Move the repository to a network block volume, and the second machine can’t mount it at all. Not one of these raises an error. Each comes from a choice the storage makes about sharing, and vendors choose differently. This chapter shows each failure, then what the three common ways agents share files need from storage. ### Lost updates **Two saves, one change lost** Agents A and B each read `plan.md`, which has three lines. A changes line 1 and B changes line 3. A saves first. Then B saves its whole copy, which still has the old line 1. Both saves return success. The file ends up with B’s new line 3 and the old line 1, so A’s change is lost, and neither agent sees an error. An agent saves a file by writing out its own copy, and that copy came from a read made before the other agent’s edit. Storage can’t merge the two, because it never sees edits, only bytes. Shared sandbox volumes call this rule “last write wins”: Modal’s documentation uses those words, and Daytona documents its Volumes the same way. NFS behaves the same, unless every program takes a lock first. What storage can do is refuse the second writer. Then the agent gets an error, reads the file again, and reapplies its edit. Two saves at the same moment can also do worse than lose an edit: their writes can interleave and leave a file that matches neither version. [How it works](https://archil.com/guides/how-file-systems-work/one-file-many-caches#writes-are-bytes-at-offsets) shows why. ### Stale reads We created a file on one machine and timed how long a second machine took to see it, on six setups. **Until another machine sees a new file** We created a file on one client and measured how long a second client took to see it. Archil, when the reader runs `archil invalidate-cache` first: about 70 ms. Mountpoint for Amazon S3: 284 ms, with no metadata cache by default. A Daytona Volume: about 360 ms, the median of 10 runs. Amazon S3 Files: 30 s, the length of its NFS directory attribute cache. Archil with default client caching: 30 s, its client cache expiry. A Modal Volume: not within 60 s, because a running sandbox sees another sandbox’s commit only after it reloads the Volume. The slow results aren’t slow networks. Every client caches what it has looked up, so it can answer `stat` and `open` without asking a server. It keeps each entry for a set time and answers from memory until that time runs out. Amazon S3 Files mounts over NFS, and the Linux NFS client keeps a directory’s attributes for at least 30 s by default. (The `acdirmin` default, from the [nfs(5) man page](https://man7.org/linux/man-pages/man5/nfs.5.html).) Archil’s client keeps entries that long by default as well. Mountpoint keeps no metadata cache by default, so it asks S3 every time, at the cost of a round trip per lookup. A Modal sandbox sees another sandbox’s commit only after it reloads the Volume, which works only when no files are open. For an agent, the window decides what a reader can trust. A reviewer that looks too soon tests old code. A coordinator that polls for `done.json` waits out the cache before it sees the file. Ask how a reader gets the current state on demand. On Archil, the reader runs `archil invalidate-cache` on the mount before it reads. [How it works](https://archil.com/guides/how-file-systems-work/one-file-many-caches) covers how caches learn that they’re stale. ### Volumes that attach once A network block volume avoids every problem so far by refusing to share: an EBS gp3 volume attaches to one instance at a time. That isn’t an arbitrary limit. The file system on the volume, such as ext4 or XFS, keeps its map of which blocks belong to which files in one kernel’s memory, and a second kernel writing the same device would overwrite that map. (EBS Multi-Attach lets one `io1` or `io2` volume attach to up to 16 Nitro-based instances in one availability zone. AWS’s documentation says standard file systems such as XFS and EXT4 “are not designed to be accessed simultaneously by multiple servers”. For more information, see [Multi-Attach](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes-multi.html).) **One volume, one machine** Machine B asks to attach the volume while machine A holds it, and the attach is refused. Only after A unmounts and detaches the volume can B attach it. The volume never belongs to two machines at once. A handoff is a sequence: unmount, detach, attach, mount. That works for an agent that resumes later on another machine. A reviewer that wants to read while the writer works can’t do it at all. ### Three ways agents share Agent systems share files in three common patterns, and each needs something different from storage. **Three ways agents share files** Three patterns. In a parallel fan-out, five attempts at one task each work on their own branch of the workspace, and the one that works is kept. Beside a writer, one agent writes and several reviewers read the same files. In a handoff, a sandbox stops partway through a task, and a new sandbox mounts the same files and continues from where it stopped. - **A parallel fan-out** runs several attempts at one task. The attempts don’t need to share a writable directory, and sharing one invites every failure in this chapter. They need isolation: a copy per attempt, so you can keep the one that worked. On Archil-managed disks, a checkpoint and a branch take 13 ms and 33 ms at the median, whatever the disk’s size. For more information, see [Can you fork it?](https://archil.com/guides/evaluating-storage/can-you-fork-it) - **A reviewer beside a writer** is one writer and many readers. The writer needs nobody else writing its files. The readers need a known limit on how stale a read can be, and a way to get the current state on demand. - **A handoff** moves a task from one sandbox to the next, often because the first one timed out. The new sandbox needs every write the old one covered with `fsync()`. It also needs the old sandbox’s write access to end, including when that sandbox died without releasing it, at the cost of no more than the writes it never sent. ### Archil’s mount modes Any number of Archil clients can write one disk, and you choose how they coordinate. In conditional mode, several clients write the same paths at once, and each write runs on our servers, as it would over NFS. When one client does most of the writing in a path, it can check that path out instead. While it holds the checkout, other clients read but don’t write there. (Our docs call a checkout a delegation. For more information, see [Sharing disks](https://docs.archil.com/concepts/sharing-disks).) The holder answers `open`, `write`, and `rename` from memory, like a local disk, and sends its changes to our servers about once a second and on every `fsync()`. How much of this you manage depends on the mode you mount in: - **Exclusive mode**, the default, checks out the whole disk for one client. Other clients can mount it with `--read-only`. - **Shared mode** (`--shared`) leaves the disk read-only on each client until it runs `archil checkout `. Opening a file for writing, or creating one, checks out that file automatically. Deleting or renaming needs the parent directory checked out. - **Conditional mode** (`--conditional`) lets a client write anywhere that nobody has checked out. Creates, renames, and deletes run on our servers before they return, and the last write to reach the disk wins. Its performance is that of a remote file system. **What client B can do, by mode** Client A has checked out `/repo` (in exclusive mode, the whole disk). In exclusive mode, B mounts with `--read-only`: it can read everything, and every change fails as read-only until A unmounts. In shared mode, B can read, can’t change `/repo/plan.md` until A checks it in, can write a file nobody holds (opening it checks it out until B closes it), can’t delete a file until it checks out the parent directory, and writes at local speed after `archil checkout /data`. In conditional mode, B can read, changes and deletes outside `/repo` run on the server at remote-file-system speed with the last write winning, and a write to `/repo/plan.md` can return and then fail when it’s sent: B’s mount turns read-only and its unsent writes are lost. The modes don’t mix freely. If another client checks out a path that a conditional client is writing to, or mounts the disk in exclusive mode, the conditional client’s mount turns read-only, and the writes it hadn’t sent are lost. (For more information, see [Writes fail on a conditional mount](https://docs.archil.com/reference/troubleshooting#problem-writes-fail-on-a-conditional-mount).) For the three patterns: - **Fan-out:** give each attempt its own branch, mounted exclusive. - **Reviewer:** mount the writer exclusive and the reviewers `--read-only`. Have each reviewer run `archil invalidate-cache` before it reads. - **Handoff:** the new sandbox mounts once the old one has unmounted. If the old one died while holding the disk, `archil mount --force` or `archil checkout --force` takes over. That loses whatever the old client hadn’t sent, the way force-detaching an EBS volume does. ### Two tests to run on any shared storage You can test both failures in a few minutes with two machines that mount the same storage at `/mnt/shared`. First, visibility. Start the loop on machine B, then create the file on machine A, and compare the two timestamps: ```bash ## Machine B: cache the directory, then wait for the file to appear. ls /mnt/shared > /dev/null until [ -e /mnt/shared/probe ]; do sleep 0.05; done; date +%s.%N ## Machine A: touch /mnt/shared/probe; date +%s.%N ``` The difference, with both clocks synced by NTP, is how long a reader can be stale. Next, lost updates. Run this on both machines at the same moment, then count the lines: ```bash for i in $(seq 1000); do echo "$(hostname) $i" >> /mnt/shared/log; done wc -l /mnt/shared/log ``` Fewer than 2,000 lines with no error on either machine means the storage lost writes silently. On NFS, `O_APPEND` is simulated by the client, and the Linux `open(2)` man page warns that it “may lead to corrupted files” when several machines append at once. ### Where Archil falls short Readers don’t see another client’s changes until their cached entries expire, unless they run `archil invalidate-cache`. Checkouts are managed by hand today: your code decides who checks out what, and when to check it in. Conditional mode lets many clients write the same files, but at the speed of a remote file system, and a conditional client loses its unsent writes if another client checks out the path it’s writing to. ## 5. Can you fork it? Source: https://archil.com/guides/evaluating-storage/can-you-fork-it · Part of Evaluating storage for agents · Facts checked 2026-11-04 **The short answer:** An agent harness forks the workspace for every attempt and every undo. If a fork is a copy, each one costs time and storage in proportion to the workspace. A copy-on-write fork is ready in milliseconds at any size and stores only what each attempt changes, as long as it includes the writes your program just made. Your harness is about to try five fixes for a failing test, each in its own copy of the workspace. The workspace is 100 GiB: a repository, its dependencies, build caches, and a test dataset. If your storage makes a fork by copying, each attempt waits while 107 GB is read and written again. At 500 MB/s, a brisk rate we assume here rather than one we measured, that's about 215 seconds per attempt before the agent runs a single command. Copy-on-write storage makes the same fork without copying the data. On Archil, a checkpoint takes 13 ms and a branch 33 ms at the median, whatever the size of the disk. (The p95s are 19 ms and 36 ms, measured with our CLI from a c7gn.2xlarge in AWS `us-east-1`. The branch time includes opening a new TLS connection. For more information, see [Limits](https://docs.archil.com/reference/limits) and the [latency benchmark](https://docs.archil.com/reference/benchmarks/latency) in our docs.) The rest of this chapter is about what that difference does to a harness, how to tell which kind of fork a vendor offers, and the one property that makes a fork trustworthy. **Copy or branch** A full copy and a branch of a 100 GiB workspace start at the same moment, played in real time. After 5 seconds, the copy, at an assumed 500 MB/s, has written 2,500 MB of about 107,000 MB and has about 3.5 minutes to go. The branch was ready in 33 ms, and the agent has been working in it ever since. ### What a copy costs a harness Harnesses fork in two patterns. The first is fan-out: take one checkpoint, branch it once per attempt, keep the attempt that passes, and throw the rest away. Each attempt sees a workspace that no other attempt can touch. **Five attempts from one checkpoint** The harness takes one checkpoint of the workspace and makes five branches from it, one per attempt. Each agent applies a different plan in its own branch and runs the tests. Attempt 4 passes, and its branch is kept. The other four fail, and their branches are thrown away. No attempt copied the workspace. The second is undo. A file system commits work one system call at a time, but an agent's unit of work is the turn. A `git clone` or an `npm install` is thousands of file operations, and a turn that fails halfway leaves half of them behind. If the harness checkpoints after each turn that succeeds, it undoes a bad turn by branching from the last good checkpoint. **Checkpoint every turn** An agent's harness takes a checkpoint after each turn that succeeds: turn 1 runs `git clone`, turn 2 runs `npm install`. Turn 3 edits the source and breaks the tests. Instead of repairing the damage, the harness branches from the checkpoint after turn 2 and runs turn 3 again there, where the tests pass. The failed turn's writes stay behind on the old line. With copies, both patterns cost time and storage for every fork. Sixty-four attempts on the 100 GiB workspace means 64 × 215 seconds, about 3.8 hours of copying, and 6,400 GiB of storage that is almost all identical. Running the copies in parallel only divides the same bytes across the bandwidth you have. A harness that pays that much forks less: it runs fewer attempts and checkpoints less often. Try your own numbers in the following calculator. The storage figures for branches follow our metering rule: data that branches share is billed once, plus what each branch adds. (For the rule and a worked example, see [Metering](https://docs.archil.com/concepts/metering#branches-and-checkpoints) in our docs. Archil bills active storage at $0.20 per GiB-month.) **What a fan-out costs** The calculator compares full copies with copy-on-write branches for any number of attempts, disk size, and share of the disk each attempt changes. Copies cost the whole disk per attempt in time and in storage. Branches cost one 13 ms checkpoint plus 33 ms per branch, and store the base once plus each attempt's changes. With the example from our metering docs (50 attempts on a 100 GiB disk, each changing 5 GiB), copies take about 3 hours and store 5,100 GiB. Branches take about 1.7 seconds and store 350 GiB. ### How a fork can cost nothing A file system already keeps a map from each file to the blocks that hold its data. A copy-on-write fork doesn't copy the blocks. It starts a second map that points at the same blocks, so the work is the same at 1 GiB as at 10 TiB. When either side later writes, the storage puts the new data in a new block and repoints only that side's map. The two sides share every block that neither has changed. **Shared blocks, then one new one** A disk's map points at four blocks, `A` to `D`. A checkpoint freezes the map. A branch starts with its own map pointing at the same four blocks, so no data moves. When the branch rewrites `C`, the storage writes a new block, `C′`, and repoints only the branch. The parent still reads `C`, and the two keep sharing `A`, `B`, and `D`. Three things follow that you can use when you evaluate a vendor. A fork whose time grows with the size of the disk is a copy, whatever it's called. Storage for a fork grows with what the attempt changes, so an attempt that rewrites the whole workspace costs as much as a copy. And deleting a fork is harder than making one, because the storage has to work out which blocks no other fork still uses. Ask whether you can delete forks, and when their space comes back. ### A fork is only as current as the storage A fork captures whatever the storage holds at that moment. Storage that acknowledges `fsync()` from a local drive and uploads the data later makes its forks from the uploaded copy, which lags. We tested this on Vercel Drives. A sandbox wrote a 4 KiB file and a 64 MiB file, called `fsync()`, and then the script forked the Drive and read the fork from a new sandbox. (The script and every result are in [Vercel Drive fsync durability](https://docs.archil.com/reference/benchmarks/vercel-drive-durability) in our docs. For why a write acknowledged from a local drive isn't safe yet, see [When is a write safe?](https://archil.com/guides/evaluating-storage/when-is-a-write-safe)) The following table shows what the forks held, by the time between `fsync()` returning and the fork request. | Time after `fsync()` returned | Forks missing the new file | | ----------------------------- | -------------------------- | | 0 s | 11 of 11 | | 1 s | 8 of 8 | | 5 s | 9 of 10 | | 30 s | 9 of 10 | | 60 s | 0 of 10 | A harness that forks the moment a turn ends gets a workspace that is missing the turn. If the turn was an `npm install`, the fork holds part of one. Vercel's docs describe a Drive fork two ways: as "copying the drive's data", and as a drive that inherits its parent's data "but stores its own changes independently". We didn't time Drive forks, so we can't say which one you pay for. The first test in the following section finds out whether a fork's time grows with the disk. (Both phrases are from Vercel's [Drives](https://vercel.com/docs/sandbox/concepts/drives#forking-a-drive) documentation.) ### Testing a fork yourself Two tests tell you most of what you need. Run them on a scratch disk with the vendor's own fork command in place of `fork`. ```bash ## 1. Is it a copy? Time a fork of a small disk and of your largest workspace. time fork small-disk try-small # 1 GiB time fork big-disk try-big # your largest workspace ## 2. Does it include what fsync() made safe? Write, sync, fork at once, compare. dd if=/dev/urandom of=/work/probe bs=1M count=64 conv=fsync sha256sum /work/probe fork big-disk try-probe # immediately ## Mount try-probe on another machine, then: sha256sum /mnt/try-probe/probe # must match, every time ``` Run the second test 10 times. One mismatch means the storage forks from a copy that lags its acknowledgments. Then delete the forks and watch whether the storage you're billed for goes down. ### Forks elsewhere The following list covers how other kinds of storage fork. - **Local copy-on-write file systems**, such as btrfs and ZFS, snapshot and clone in moments. The fork lives on the same disk, so no other machine can mount it. - **Amazon EBS** stores incremental snapshots in Amazon S3. A volume restored from one fetches its blocks from S3 as they're read, and can run with higher latency until every block has arrived. (From AWS's [Amazon EBS snapshots](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-snapshots.html) and [Initialize Amazon EBS volumes](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-initialize.html) pages.) - **Amazon EFS** has no native branching. You copy the data. - **JuiceFS** clones a file or directory by copying only its metadata. A directory clone isn't atomic if the source changes during it. (From JuiceFS's [clone documentation](https://juicefs.com/docs/community/guide/clone/).) - **Tigris** forks buckets without copying data. A fork is a bucket, so an agent that runs git or npm needs a file system on top of it. ### What Archil does In AWS regions, every Archil-managed disk can checkpoint and branch, and Archil-managed disks are the default. A branch is a disk in its own right. You can mount it on any machine, including one other than its parent's, and run two branches of a disk on separate machines at once. End to end, checkpointing a disk, branching it, and mounting the branch took about 0.9 s from inside a Vercel sandbox and about 1 s from a Modal sandbox. A 1 GiB file was intact in the branch both times. ```bash archil checkpoints create /mnt/archil before-attempts archil branches --region aws-us-east-1 create dsk-0123456789abcdef attempt-1 \ --from-checkpoint before-attempts sudo archil mount myorg/mydisk /mnt/attempt-1 \ --region aws-us-east-1 --branch attempt-1 ``` Before you take a checkpoint, call `fsync()` on the files the turn wrote. Archil's `fsync()` returns once the write is stored redundantly across Availability Zones. Our sandboxes fork the same way. `sandbox.fork()` pauses a running sandbox, checkpoints its root disk, branches it, and resumes the original. The fork starts from the original's exact state. **Forking a running sandbox** When you call `sandbox.fork()` on a running Archil sandbox, the sandbox pauses. Its root disk is checkpointed in about 13 ms and branched in about 33 ms, and the original resumes. The fork starts from the original's exact state, on its own branch. Archil's forks fall short in four ways today: - **Archil-managed disks only.** A disk that syncs to your own bucket can't checkpoint or branch. - **No deleting.** You can't delete an individual branch or checkpoint, so the data an abandoned attempt wrote stays on the disk. If you fork thousands of times a day, plan for that until we ship deletion. - **No merging.** Branches diverge and don't merge back into their parent. To keep an attempt, you carry on from its branch. - **Not a security boundary.** Credentials cover a whole disk, every branch and checkpoint included. Give each untrusted tenant its own disk. A sandbox with other disks mounted must be stopped before it forks, and the fork shares those disks rather than branching them. For how copy-on-write works underneath, block by block, and why deleting a fork is the hard part, see [Copy-on-write, block by block](https://archil.com/guides/how-file-systems-work/copy-on-write). ## 6. What does it cost when nothing is running? Source: https://archil.com/guides/evaluating-storage/what-does-it-cost-when-nothing-is-running · Part of Evaluating storage for agents · Facts checked 2026-11-04 **The short answer:** An agent’s workspace sits idle most of the time it exists, so the bill depends on what storage charges for while nothing runs. A provisioned volume bills its full size every hour, and a sandbox’s own disk costs whatever the machine costs to keep up. Archil bills its active rate only for data used in about the last hour, plus a low rate on the whole disk. Give each of 1,000 agents a 100 GiB EBS gp3 volume, and you pay $8,000 a month before any agent runs a command. Keep a sandbox up for each one so its local disk survives, and the compute adds about $121,000 a month. Most of those hours, the agents are waiting. This chapter measures how much of an agent’s life is waiting, using our own team’s agents, and then prices what each kind of storage charges for it. (Every price here is a list price in us-east-1 (N. Virginia), before discounts, free tiers, and taxes. AWS bills a GB as 2³⁰ bytes, the same as a GiB, so this chapter applies AWS’s per-GB rates per GiB. For how providers meter storage and why their prices take the shape they do, see [How storage is metered](https://archil.com/guides/how-file-systems-work/how-storage-is-metered).) ### Where an agent’s time goes We read the timestamps on 30 days of our engineering team’s coding-agent sessions: 384 agent loops and about 37,000 tool calls. We read only when each model turn and tool call started and ended, not what they contained. The median tool call finished in 0.5 s, and 89% finished in under 10 s. About 1% ran longer than 5 minutes, such as builds and test suites, and that 1% accounted for about half of all tool time. Between calls, the agent waits for the model to write its next step, and often for a person to read the result. Suppose each loop had started a sandbox at its first tool call and stopped it after 15 minutes without one. That sandbox would have been idle for 65% of the time it ran, and for 88% in the median loop. A 5-minute auto-stop still leaves it idle 48% of the time, and it starts more often. (The method and results are in [Do AI agents need a sandbox running for the whole session?](https://archil.com/article/do-ai-agents-need-a-sandbox-running-for-the-whole-session)) **One agent loop and its sandbox** An example agent loop over two hours: three bursts of short tool calls and one 6-minute build, about 7 minutes of tool work in all, separated by waits on the model and a 45-minute wait on a person. A sandbox that stops after 15 minutes without a tool call is up for 73 minutes, almost all of it idle. One that stops after 5 minutes is up for 51 minutes but starts three times. These numbers belong to the example. The measured idle shares are in the text. ### What each kind of storage bills for the waiting A workspace has to outlive every tool call and usually every session, so its storage bills through all of that waiting. What it bills for depends on the model: - **A provisioned volume** bills its size. EBS gp3 costs $0.08 per GB-month, charged, in AWS’s words, “by the amount of GB you provision”. A full volume and an empty one cost the same, and EBS can grow a volume but can’t shrink it, so each volume carries headroom for the largest workspace you expect. (From [Amazon EBS pricing](https://aws.amazon.com/ebs/pricing/) and [Modify an Amazon EBS volume](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-modify-volume.html): “You can’t decrease the volume size.”) - **A sandbox’s own disk** bills as the machine. To keep the files, you keep the sandbox running, or pay the vendor to keep a stopped copy. An idle sandbox with 2 vCPUs and 4 GiB costs about $0.17 an hour at E2B’s list price, $121 over a 730-hour month. (E2B bills $0.000014 per vCPU-second and $0.0000045 per GiB-second ([E2B pricing](https://e2b.dev/pricing)). Other providers’ idle hours are in [What does an idle agent sandbox cost per hour?](https://archil.com/article/what-does-an-idle-agent-sandbox-cost-per-hour)) - **Storage billed by what it holds** charges for bytes stored, so an empty workspace costs nothing. EFS Standard costs $0.30 per GB-month and S3 Standard $0.023. EFS moves files to cheaper classes after 30 days without access, but a workspace the agent opens every day restarts that clock. (From [Amazon EFS pricing](https://aws.amazon.com/efs/pricing/), [EFS lifecycle management](https://docs.aws.amazon.com/efs/latest/ug/lifecycle-management-efs.html), and [Amazon S3 pricing](https://aws.amazon.com/s3/pricing/).) - **Archil** bills $0.20 per GiB-month of active data: data read in about the last hour, or written and flushed in about the last hour. We measure it every minute and bill the time-weighted average. Reading metadata with `ls`, `stat`, or `find` doesn’t make file data active. The whole disk also bills at $0.025 per GB-month on a disk we manage, or at your bucket’s own rate on a disk synced to your bucket. (From [Metering](https://docs.archil.com/concepts/metering) and [Pricing & Billing](https://docs.archil.com/administration/billing). Branches and checkpoints are copy-on-write, so data they share bills once.) **Four meters, one day** The figure follows one agent workspace through a day with three working sessions, about 4 hours of tool calls in all. A sandbox kept running bills all 24 hours, and a provisioned volume bills its full size for all 24. Archil’s active meter counts the data the agent touched, during each session and for up to an hour after, about 7 hours in all. All day, the whole disk also bills at the archive rate, or at your bucket’s rate on a disk synced to your own bucket. ### Pricing a fleet Here’s the arithmetic for one workspace that holds 5 GiB, whose agent works 2 hours a day in one session and touches half its data. The active meter runs about 3 hours a day: the session, plus the hour after it. That’s 5 GiB × 50% × 3/24 × $0.20, about $0.06 a month. The archive rate on the whole disk, about 5.37 GB × $0.025, adds $0.13. The workspace costs about $0.20 a month, against $1.60 for a 20 GiB gp3 volume or $1.50 on EFS Standard. The calculator does the same arithmetic for a fleet and splits each bill into the hours the agent works and the hours it waits. **Price a fleet of workspaces** The calculator starts with 1,000 workspaces of 5 GiB each on 20 GiB volumes. Each agent works 2 hours a day in one session and touches half its data. At those settings, EBS gp3 costs $1,600 a month, EFS Standard $1,500, and S3 Standard alone $115, and about 92% of each falls in the hours the agent waits. Archil costs about $197 a month on managed disks and $178 synced to your own S3 bucket. A sandbox kept running for each workspace so its disk survives adds about $120,900 a month in compute. One that stops after 15 idle minutes adds about $11,300, before whatever the vendor charges to keep its stopped copy. It leaves EFS in Standard, before throughput charges. The calculator makes one conservative assumption about Archil: it bills the archive rate on every byte, all month, on top of the active rate. That reading gives the highest bill. If archive applies only to inactive data, a managed disk costs slightly less than shown. ### Where a volume costs less Our active rate is 2.5 times gp3’s, so a workspace that’s busy all day costs more on Archil than on a volume sized to fit it. If an agent touches all 5 GiB of its workspace around the clock, that’s $1.00 a month active plus $0.13 archive, against $0.40 for a 5 GiB gp3 volume. With one session a day, a tightly sized gp3 volume costs less once the agent touches all its data for more than about 5 hours a day. Data that nobody touches costs less still in plain object storage, which is why a disk synced to your bucket leaves cold data at your bucket’s rate. ### Compute that stops between calls The same reasoning applies to the machine. A serverless tool call runs one command in a short-lived microVM with the disk mounted, returns the output, and shuts down. Between calls, nothing runs and no compute bills. **A serverless tool call** The agent’s harness sends `npm test` to the disk. A microVM starts with the disk mounted, runs the command, returns its output, and shuts down. The call bills $0.00005. Until the next call, nothing runs and no compute bills, and the files stay on the disk. We charge $0.05 per 1,000 calls, with execution included. Our agents made about 160 tool calls per hour of activity, so 2 hours a day for a month is about 9,700 calls, or $0.49. Per-call pricing has limits. At about 3,300 calls an hour, close to one a second sustained, it costs the same as an idle E2B sandbox of the same size. Each call can run for up to 5 minutes, and a dev server or a Python kernel has to stay up between calls. For those, use a persistent sandbox with the same disk mounted. (From [Serverless execution](https://docs.archil.com/compute/serverless-execution), [Sandboxes](https://docs.archil.com/compute/sandboxes/introduction), and our [pricing page](https://archil.com/pricing). Each call gets 2 vCPUs and 4 GiB. Our persistent sandboxes bill $0.055 per vCPU-hour and $0.020 per GiB-hour while they run.) ### Measuring your own idle time Before comparing prices, measure how much of your agent’s time is waiting. Log when each tool call starts and ends, as one JSON object per line, for one agent loop. The following script replays those calls against a sandbox that stops after 5 or 15 minutes without a call, and prints how much of its running time it would sit idle: ```python ## idle_share.py < calls.jsonl, one {"start": ..., "end": ...} per line, in Unix seconds import json, sys calls = sorted((c["start"], c["end"]) for c in map(json.loads, sys.stdin)) def idle_share(stop_after): running = busy = 0.0 up = done = calls[0][0] # the sandbox starts with the first call for start, end in calls: if start > done + stop_after: # idle too long: it stopped, and starts again running += done + stop_after - up up = start busy += max(0.0, end - max(start, done)) done = max(done, end) running += done + stop_after - up return 1 - busy / running for minutes in (5, 15): print(f"{minutes}-minute auto-stop: idle {idle_share(minutes * 60):.0%} of running time") ``` Then put your own numbers in the calculator. To check a vendor’s bill, create one workspace, leave it untouched for 48 hours, and read every line item that kept accruing. ## 7. Where does your data live? Source: https://archil.com/guides/evaluating-storage/where-does-your-data-live · Part of Evaluating storage for agents · Facts checked 2026-11-04 **The short answer:** Where an agent’s files end up decides whether your other tools can read them, whether leaving a vendor is a copy or a migration, and what you stay tied to. Ordinary objects at their paths, in a bucket you own, keep all of that open; chunked formats, disk snapshots, and a vendor’s storage don’t. Archil’s default managed disk uses our internal format, so files reach your own bucket only on a disk you sync to it. An agent finishes a run and leaves `data/events.parquet` behind. If that file is an ordinary object in your own bucket, an analyst can query it with DuckDB that afternoon, from a laptop that has never heard of your agent platform. If it’s numbered blocks that only one client can reassemble, or a snapshot in a vendor’s account, the analyst needs a mount, an export job, or a ticket first. The same difference comes back the day you leave a vendor. With ordinary objects in your bucket, leaving means pointing the next system at the bucket. With anything else, it’s a migration: restore, mount, copy, and check. “Your data is in S3” doesn’t tell you which case you’re in. The following figure shows what one agent workspace leaves in object storage on each kind of storage. Pick one to see what your bucket holds, what other tools can read, and how you’d leave. **What your bucket sees** The figure lists one agent workspace (a Git repository, a shell script with mode 755, a symbolic link, an installed package, a Parquet file, and a SQLite database) as each kind of storage leaves it in object storage. An Archil-managed disk, the default, leaves nothing in your bucket: it’s stored in Archil’s internal format, in a bucket Archil operates, and you list it through the disk’s own S3 API, where the symbolic link isn’t readable. A bucket-synced Archil disk leaves every file as an ordinary object at its path; the script’s mode stays in Archil, and the symbolic link becomes a text file holding its target. Mountpoint for Amazon S3 leaves ordinary objects too, but can’t write the Git repository, the symbolic link, or the SQLite database, and can’t change modes. Amazon S3 Files leaves ordinary objects as new versions, keeps the mode as object metadata, and stores the link as an object whose encoding AWS doesn’t document. JuiceFS leaves numbered block objects under `chunks/`, with names and modes in its metadata engine. A block volume leaves a snapshot you can’t read as files, and a sandbox vendor’s disk leaves nothing in your bucket at all. ### Other tools reading the files Three things decide whether a tool you already run can read an agent’s files: the format, the account, and the key. Only one combination works without help: ordinary objects, at their paths, in an account you control. DuckDB can query `s3://acme-agents/data/events.parquet` where it sits, and so can Spark, Athena, a training job, or `aws s3 cp`. The other combinations are common. AWS stores EBS snapshots in S3, “in S3 buckets that you can’t access directly.” (From the [Amazon EBS snapshots guide](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-snapshots.html).) JuiceFS writes to your own bucket in its own format, and its docs say that once you’ve written a file, “you cannot find the original file directly in the object storage.” (From the [JuiceFS architecture page](https://juicefs.com/docs/community/architecture). Its S3 gateway can serve the files over the S3 API, but that’s JuiceFS reading them for you.) Sandbox vendors keep workspaces on their own storage, in their own accounts. Fly.io backs a Sprite’s disk up to object storage when the Sprite goes idle, and Vercel keeps a sandbox’s file system snapshot for 30 days after last use by default. Each is a reasonable product. Each also turns the sandbox into a separate data island that you copy into, and back out of. **A data island** An agent commits in a vendor’s sandbox, and its files are saved to the vendor’s storage, in the vendor’s account. Your bucket stays empty, and your own tools have no access to the vendor’s storage. The files become usable only after a copy you schedule through the vendor’s API, and that copy is out of date as soon as the agent writes again. Ordinary objects in your bucket still don’t carry everything a file system holds. An object has no mode bits and no links, and S3 has no rename, so every product that writes ordinary objects decides what to give up. Mountpoint for Amazon S3 gives up what it can’t write: it writes files sequentially, supports neither symbolic nor hard links, and renames files only in S3 Express One Zone. (From Mountpoint’s [semantics document](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md).) That suits training data. It doesn’t suit an agent, because `git` renames files into place and SQLite rewrites pages in the middle of a file. For more information about how each design maps files onto objects, see [How it works](https://archil.com/guides/how-file-systems-work/a-file-system-on-an-object-store). ### Leaving: a copy or a migration Leaving is where the format turns into time and money. The following list shows what leaving involves for 500 GB of agent workspaces: - **Ordinary objects in your bucket** (a bucket-synced Archil disk, Mountpoint, or S3 Files): the files are already there. Stop writing, let any sync finish, and point the next system at the bucket. - **An Archil-managed disk:** copy it out over the disk’s S3 API, or mount it and copy it with `rsync -aH` to keep owners and links. Transfer out of Archil costs $0.07 per GB, so 500 GB × $0.07 = $35. - **JuiceFS:** mount the volume with JuiceFS, with its metadata engine still running, and copy every file out through it. - **A block volume or a sandbox vendor’s disk:** restore and attach the volume, or export through the vendor, and then copy. **Leaving with 500 GB** Moving 500 GB of agent workspaces somewhere else. On a bucket-synced disk, the files are already ordinary objects in your bucket, so the next system points at the bucket, nothing moves, and leaving costs $0 in transfer. An Archil-managed disk is copied out over its S3 API at $0.07 per GB, $35 for 500 GB. A JuiceFS volume has to be mounted and every file read back through the JuiceFS client and its metadata engine. ### What stays behind Even the best case carries less than the file system held. The following table shows what reaches your bucket from the three products in this chapter that write ordinary objects. | | Mountpoint for Amazon S3 | Amazon S3 Files | Archil, bucket-synced disk | | ------------------- | -------------------------------------------- | -------------------------------------------------- | ----------------------------------------------------- | | File contents | An object when the file is closed or flushed | A new object version after 60 s without writes | An object, tens of seconds to minutes after `fsync()` | | Mode bits and owner | Fixed at mount; `chmod` isn’t supported | Stored as user-defined object metadata | Kept in Archil, not in the bucket | | Symbolic link | Not supported | Stored as an object; the encoding isn’t documented | A text file holding the target | | Hard link | Not supported | Not supported | A full copy | | Directory rename | Not supported | Copied and deleted object by object | Partial results visible in the bucket while it runs | So an S3 tool reading a bucket-synced Archil disk sees a shell script without its executable bit, and a symbolic link as a short text file. A system you point at that bucket later inherits the contents, not the modes or the links. (Sources: Mountpoint’s [semantics document](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md); AWS’s guide to [S3 Files synchronization](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-files-synchronization.html) and its [limits page](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-files-quotas.html); our [data sources](https://docs.archil.com/concepts/data-sources) page.) The other formats tie you more tightly. A JuiceFS bucket is readable only with JuiceFS and the metadata engine you run beside it, so losing the engine loses the map to every file. A block snapshot is readable only after a machine in that provider’s cloud attaches it. A sandbox vendor’s storage lasts as long as the vendor’s retention rules. ### Two kinds of Archil disk You choose the format when you create an Archil disk. The default, an **Archil-managed disk**, stores data in Archil’s internal format, in a bucket that we operate. That format is what lets us checkpoint a disk in 13 ms and branch it in 33 ms at the median, whatever its size. You can’t point your own tools at that bucket. You read the disk through its own S3 API, which is strongly consistent with the file system, or through a mount, and both are served by us. (For more information, see [Data ownership and portability](https://docs.archil.com/reference/data-ownership) and the [S3 API](https://docs.archil.com/protocols/s3-api) in our docs. For checkpoints and branches, see [Can you fork it?](https://archil.com/guides/evaluating-storage/can-you-fork-it)) A **bucket-synced disk** points at your bucket in S3, GCS, R2, or another S3-compatible store, and writes each file there as an ordinary object at its path. A write is safe once `fsync()` returns, because by then it’s replicated across availability zones. The bucket receives it later, in the background, and you can’t force that sync. Deleting the disk leaves the objects in your bucket, and removing the credentials you granted cuts us off. **Safe now, in your bucket later** The agent writes `events.parquet` and calls `fsync()`. Archil replicates the write across availability zones, and then `fsync()` returns: the data is safe. Tens of seconds to minutes later, the object lands in your bucket at `data/events.parquet`, where DuckDB on any machine queries it with no Archil client or account. Two limits are worth stating plainly. The disk you start on by default is the one your tools can’t read without us. And when another program writes a file in the bucket at the same time that an agent writes it through the disk, the result is undefined and can include lost or corrupted data. S3 Files has a rule for that case: the bucket wins, and the file system’s version moves to a lost-and-found directory. We don’t, so write different paths from each side, or send those writers to the disk’s own S3 API. ### A test you can run A vendor’s architecture diagram won’t show you what lands in the bucket. A five-minute test will. On the vendor’s storage, build a small tree that uses what agents rely on: ```bash mkdir -p check/sub check/empty && cd check printf 'hello\n' > a.txt && printf 'x\n' > sub/f.txt chmod 755 a.txt ln -s a.txt link.txt ln a.txt hard.txt mv sub renamed sync ``` Wait as long as the vendor says a sync takes. Then, from a machine with none of the vendor’s software and only your own credentials, look at the bucket: ```bash aws s3 ls --recursive s3://your-bucket/check/ aws s3 cp s3://your-bucket/check/a.txt - aws s3api head-object --bucket your-bucket --key check/a.txt ``` Check each file against the earlier table. Is `a.txt` there, under its own name, and does `head-object` show its mode anywhere? Is `link.txt` missing, a text file, or something else? Is `hard.txt` a second copy? Did `empty/` survive, and is anything left under `sub/`? If the listing is empty, or full of numbered blocks, your tools can’t read these files without the vendor. Then ask for the exit in writing: if you cancel tomorrow, which files remain, in what format, and in whose account.