When fsync returns, POSIX says the data has reached stable storage, so a reader that comes to that storage afterwards should find it. Sandbox storage doesn’t always work that way: some storage acknowledges fsync before the write reaches the copy that other sandboxes, snapshots and forks read. A fast return is useful, but only if the data has reached the storage the next reader will use.
Archil’s test found that a file fsynced in one sandbox was missing from every Drive snapshot and fork taken right afterwards. An Archil disk mounted in a new sandbox immediately after fsync returned had the file in 20 of 20 trials. Those results cover only the read paths and conditions that were tested, which is why the last section shows how to run the same test on your own storage.
What fsync durability means for sandbox storage
POSIX says that when fsync returns, the data has reached stable storage, so a reader that comes to that storage afterwards should find it. An application can use the return as the boundary between a write that is durable and one still waiting to become durable. That boundary becomes especially important when the writer and reader live in different sandboxes. The writer’s own view is not enough. The data must be available from the storage independently of the machine that wrote it.
A sandbox storage test thus needs to do more than write a file and read it back in the same sandbox. That path does not answer what a newly mounted sandbox, snapshot or fork will find. Each of those paths reads the data without going through the writer, so it shows whether the handoff worked. A write can look finished to the writer while still missing from the storage view used elsewhere.
Durability is not the same as visibility
Durability means that once fsync returns, the write survives the writer’s machine failing and is readable from the storage by any path. Visibility tells whether another running client sees the change yet. That client may have cached an older view, so its next read need not reflect what is already stored durably. Treating them as synonyms makes it easy to mistake a stale reader for missing data, or missing data for a stale reader.
The reverse distinction is just as important. A storage system can acknowledge fsync from a local cache and upload the write later. In that case, the call can return quickly even though the write has not reached the storage that other readers, snapshots or forks use. The acknowledgement gets ahead of the handoff. This is why a short fsync time, by itself, does not establish durability beyond the writer’s machine.
A newly mounted sandbox checks whether the data is there for a new reader, while a client that is already mounted may also be testing how its cache refreshes. Neither result should silently stand in for the other. The same care applies to shutdown. A clean stop that gives storage time to flush is not a test of what survives an unexpected host failure.
How the Vercel Drive test works
Archil’s test starts with sandbox A mounting a Drive read-write. The sandbox writes a 4 KiB file and a 64 MiB file using dd conv=fsync, runs sync on the directory and records each file’s sha256. Those checksums provide a test of content. A file can exist and still be wrong. The two file sizes also let the test check more than a tiny write.
After a delay, the script reads the Drive through paths that do not go through sandbox A. One variant mounts drive.snapshot() in a new sandbox, and another does the same while sandbox A keeps writing other data. A third calls drive.fork() and mounts the fork. The last variant stops sandbox A, then mounts the Drive read-write in a new sandbox. That last step changes the conditions because the writer is no longer running.
Each file receives one of three results, which are ok, missing or corrupt, with corrupt meaning that its sha256 is wrong. Archil ran the test in region pdx1 and used 10 trials per cell; a few cells ran 8 or 11. The reading sandbox takes about another 0.8 s to start. Even with a delay labelled 0 s, the reader checks the file after fsync returns. There is still startup time between the acknowledgement and the read.
Vercel’s Drives docs say that a Drive snapshot reflects the Drive as it was at the moment it was mounted. That description makes the snapshot’s storage view central to interpreting the test. The test checks whether the writer’s fsynced files are in that view after the chosen delay. It does not simply ask the original sandbox to read back its own work.
What the test found
In the ordinary snapshot variant, files were missing in 10/10 trials at each of 0 s, 0.1 s and 1 s. At 5 s, 8/10 were missing, and at 30 s, 7/10 were missing. Results were 10/10 ok at both 60 s and 120 s. The gap was not confined to the immediate read. In this variant, some snapshots still lacked the fsynced file after a delay of 30 s.
In the snapshot-while-writing variant, files were missing in 11/11 trials at 0 s, 10/10 at both 0.1 s and 1 s, and 8/10 at 5 s. At 30 s, it reported 4/10 corrupt, while at 60 s it still reported 1/10 missing. At 120 s, results were 10/10 ok. Continued writing therefore belongs in the test conditions.
The corrupt copies were full size, but their last 12 to 28 MiB read as zeros. They still read that way 60 s later with the page cache dropped. The checksum showed that a copy had the expected length but the wrong contents.
The fork variant reported missing files in 11/11 trials at 0 s, 10/10 at 0.1 s, 8/8 at 1 s, and 9/10 at both 5 s and 30 s. Results were 10/10 ok at 60 s and 120 s. These results describe only the observed delays.
Stopping the writer changed the outcome substantially. After sandbox A stopped, a new sandbox mounted the Drive read-write and found the file intact in 10 of 10 trials. The Drive could be mounted again about 3 s after fsync returned. But stopping lets Vercel flush the host first, so this result does not establish what happens if the host fails.
The latency result needs that context as well. A Drive acknowledged fsync of a 4 KiB file in about 0.2 ms, which is local NVMe speed. Vercel describes Drives as offering NVMe-speed writes. Archil’s Vercel Drive fsync durability publication provides the full results and script.
What an Archil fsync means
Archil’s Data durability documentation says that an Archil fsync returns once the write is stored redundantly across Availability Zones. The distinction gives the latency a useful meaning. The latency measures the wait for redundant storage.
Archil’s Archil vs Vercel Sandbox comparison reports that creating a 4 KiB file and fsyncing it took 4.8 ms on an Archil disk in a Vercel sandbox, with the write durable on return. Both figures time the same open, write, fsync and close of a 4 KiB file. The Drive’s returns from local NVMe; Archil’s returns after the write is replicated, so the two numbers measure different guarantees.
Writes that an application never fsyncs follow a different path. Archil’s client buffers them and sends them in the background, while archil unmount waits for all of them. Unmount before a sandbox stops. The guarantee cannot cover a call the application did not make.
Other clients see a change when their caches refresh, which took about 28 s with default caching in Archil’s Vercel test. The command archil invalidate-cache makes the next read go to the server. That is different from waiting for the writer’s data to reach durable storage. A reader’s stale cache and an unfinished storage handoff may look similar from the application, but they require different explanations.
How to test your own sandbox storage
To test the handoff your application relies on, check what happens after a successful fsync. The test needs known contents, an independent read path and trials run more than once with different delays. Checksums keep the result honest.
- Write files with known content, fsync each file and its directory, and record checksums.
- Read through a path that does not go through the writer. The paths to test include a new mount on another sandbox, a snapshot, a fork, or a mount after stopping the writer.
- Sweep the delay between
fsyncreturning and the read, using values such as 0, 0.1, 1, 5, 30, 60 and 120 s. Repeat each cell several times so that one successful handoff does not stand in for the whole test. - Report ok, missing or corrupt for each file. A missing or corrupt result after
fsyncreturned means that the tested path did not see data the writer had already fsynced.
Archil’s script, vercel-fsync-durability.py, is published in full on the benchmark page and needs only the Vercel Python SDK and standard tools. It creates its own Drive and sandboxes and deletes them when it finishes, so use a dedicated Vercel project. The same approach works on other providers’ storage, because the question does not depend on a particular product name. The command below runs the snapshot, busy-snapshot, fork and stop variants. It also records the results in results.jsonl.
pip install vercel
export VERCEL_TOKEN=...
export VERCEL_TEAM_ID=team_...
export VERCEL_PROJECT_ID=prj_...
python vercel-fsync-durability.py --region pdx1 --trials 10 \
--delays 0,0.1,1,5,30,60,120 \
--variants snapshot,snapshot-busy,fork,stop --out results.jsonl