Evaluating storageChapter 2 of 7
When is a write safe?
The short answer
A write is safe once fsync() has returned and the data is stored on more than one machine. Some persistent disks for agents acknowledge fsync() from a drive inside the machine and upload the data later, so a write they called safe is lost if that machine is lost. Ask any vendor where fsync() returns from, and time it yourself.
Hunter LeathFounder & CEO, Archil
7 min readChecked November 2026Read as Markdown
We ran a test on Vercel Drives, the persistent disks that Vercel offers its sandboxes. A sandbox wrote a file and called fsync(), the system call a program makes to ask that its writes be stored permanently. As soon as fsync() returned, we took a snapshot of the Drive and opened it in a second sandbox. The file wasn’t there in any of 21 trials. Forks of the Drive taken at the same moment were missing it too.
A sandbox wrote a 4 KiB file and a 64 MiB file, called fsync() on each, and synced the directory. After a delay, a new sandbox read the Drive through a snapshot or a fork. Taken the moment fsync() returned, 10 of 10 snapshots, 11 of 11 snapshots taken while the writer kept writing, and 11 of 11 forks were missing the file. At 1 second, every snapshot and fork still lacked it. At 30 seconds, 7 of 10 snapshots and 9 of 10 forks lacked it, and 4 of 10 snapshots taken while the writer kept writing held a 64 MiB file whose last 12 to 28 MiB read as zeros. From 60 seconds on, the file was intact in every trial but one. An Archil disk mounted in a new sandbox the moment fsync() returned had the file in 20 of 20 trials.
A snapshot isn’t a crash. But a snapshot is read from the storage that the Drive would be rebuilt from if its host failed, and for most of the first minute that storage didn’t have the write. Thirty seconds after fsync() returned, some snapshots held the file with its last megabytes reading as zeros. Only from about a minute on did snapshots and forks reliably hold it.The script and every result are in Vercel Drive fsync durability. Vercel describes Drives as having “NVMe speed writes” in its Drives documentation.
If the host had died during that minute, the write would have died with it, after the application had already been told it was safe. That matters most for the software that relies on fsync() most. When a database tells you a transaction committed, it has called fsync() and waited for it to return. A user is told their account exists, or a traveler that their seat is booked. Lose the write after that, and the account is gone when they log in and the seat goes to someone else.
Machines that don’t come back§
The Linux manual page for fsync() promises that once it returns, “all changed information can be retrieved even if the system crashes or is rebooted.” The guarantee was written for a machine with a disk inside it, where the worst that usually happens is a restart.
In the cloud, the machine itself is often what fails. When AWS detects an “irreparable failure of the underlying host” for an EC2 instance, it schedules the instance to stop or terminate, and data on the host’s own drives doesn’t survive a stop or a termination.From the AWS pages on instances scheduled to retire and instance store data persistence. The same table says instance store data survives a reboot and a power failure, which is the case fsync() was designed for. A sandbox runs on whichever host has room, so its local disk is that host’s disk, and it goes wherever that host goes.
Try it in the following figure. Pick what fails, then drag the moment of the crash.
- 1.0 swrite(notes, "v1")
- 3.0 swrite(notes, "v2")
- 3.2 sfsync(notes)returns at 3.6 s
- 4.6 scrash
- 7.5 swrite(notes, "v3")
The machine never comes back: a hardware fault, a reclaimed spot instance, a sandbox host retired. Its local disks go with it.
Local disk
The machine’s own disk
nothing
lost v2
nothing
lost v2
Local disk, uploaded later
Cached here, copied to object storage after
v1
lost v2
v1
lost v2
Network block volume
Replicated on other servers, attached to one machine
v2
promise kept
v2
promise kept
Archil
Replicated across zones before fsync returns
v2
promise kept
v2
promise kept
Crash at 4.6 seconds, machine is lost. Local disk: nothing, a write fsync promised is lost; Local disk, uploaded later: v1, a write fsync promised is lost; Network block volume: v2; Archil: v2.
A program writes v1, writes v2, calls fsync(), and then writes v3. Each band shows what the program is guaranteed to read back if the crash happens at that moment, for three kinds of failure. After fsync() returns, every band should show v2 or later.
When only the program crashes, every kind of storage keeps every write, because the operating system still holds them. When the power is cut, every kind keeps the writes that fsync() covered, because the machine restarts with its disks. When the machine is lost, only storage that replicated the data to other machines before fsync() returned still has v2. The local disk loses everything, and the disk that uploads later loses v2 until the upload finishes.
A test that kills your program and then checks the file proves nothing about storage, because the operating system still has the write. A test that cuts the power passes on almost anything. The failure that separates storage products is the one you can’t easily stage: the machine never comes back. For how a write travels from write() down to the drive and out to other machines, see Where fsync returns from.
Two kinds of persistent disk§
Many sandbox platforms now market a persistent disk for agents, and they don’t all persist data the same way. There are two models, and the difference lies entirely in what happens when your program calls fsync().
In the first model, the local drive acknowledges fsync() and an upload follows. The machine is lost before the upload finishes, and the write that the application was told was safe never arrives. In the second model, the write is replicated across availability zones before fsync() returns. Losing the machine loses nothing, and the data reaches the bucket later.
The first model acknowledges fsync() once the data reaches the drive inside the machine and copies it to object storage in the background. Vercel Drives behave this way in our test. Fly.io’s documentation says a Sprite’s data “is written to fast NVMe storage” while it runs and is “backed up to durable object storage” when it goes idle.From the Sprites overview. Both companies are making a reasonable trade for speed. Benchmarks get faster, and a database’s write speed is its fsync() speed. But until the copy finishes, the write exists on one drive in one machine.
The second model replicates the data to other machines before it acknowledges, so no single machine can take the write with it. EBS replicates every volume within its availability zone, and S3 stores objects across at least three zones. Local NVMe can be safe too, with a replication layer above it: PlanetScale Metal runs databases on local drives and makes sure “every write has reached stable storage in two availability zones before it’s acknowledged to the client.”From the AWS pages on EBS features and S3 data protection, and the FAQ on PlanetScale Metal.
Archil uses the second model. Our client keeps your writes in memory, like a local file system, and when you call fsync() it sends them to our servers, which replicate them across availability zones before they acknowledge. Archil is designed for 99.999% durability for recent writes before they reach S3. Your bucket receives them later, usually tens of seconds to minutes afterward.
That safety costs time on every fsync(). A synchronous 4 KiB write on Archil takes about 2 ms at the median. A program that waits for each commit before starting the next can make at most about 500 commits per second at that speed (1 second divided by 2 ms), which is fine for an agent’s workspace and slow for a write-heavy database.Archil’s figures, with their method, are on the performance page. Writes your program hasn’t synced are still only in our client’s memory, and a power cut loses them, as it would on a local disk. Unmount with archil unmount, which flushes, rather than umount, which doesn’t.
A timing test you can run§
You can’t see where a write goes, but you can time it. Storing data on a second machine takes at least one network round trip and a write on the far side. A drive inside the machine needs neither. Run this on the storage you’re evaluating:
# About 1,000 small writes, each followed by fdatasync().
fio --name=sync --directory=/mnt/test --rw=write --bs=4k --size=4m \
--fdatasync=1 --ioengine=syncfio prints percentiles for the sync latency. Compare them with our own measurements of a similar operation in the following figure.
The median time to create a file, write 4 KiB, call fsync(), and close it, on a log scale. Two acknowledge from a drive in the same machine: a Vercel Drive took 0.18 ms and a Fly.io Sprite’s disk took 3.5 ms. Two store the write on other machines first: Archil took 4.2 ms from an EC2 instance in the same region, and Mountpoint for Amazon S3, which uploads the object before fsync() returns, took 244 ms.
The test only works in one direction. A median of a fraction of a millisecond, like the Drive’s, most likely came from a drive in the machine, although replicated block storage can come close: AWS designs io2 Block Express volumes, which replicate within their zone, for average latency under 500 microseconds.From the AWS page on EBS volume types. A slower answer proves less. The Sprite’s disk takes almost as long as Archil and still acknowledges locally.
So use the timing to decide what to ask, and ask in words a vendor can’t sidestep. When fsync() returns, which machines hold the data, and is the one running my code among them? If that machine dies a millisecond later, what does the next reader see?