# 2. When is a write safe?

Source: https://archil.com/guides/evaluating-storage/when-is-a-write-safe · Part of Evaluating storage for agents · Facts checked 2026-11-04

**The short answer:** A write is safe once `fsync()` has returned and the data is stored on more than one machine. Some persistent disks for agents acknowledge `fsync()` from a drive inside the machine and upload the data later, so a write they called safe is lost if that machine is lost. Ask any vendor where `fsync()` returns from, and time it yourself.

We ran a test on Vercel Drives, the persistent disks that Vercel offers its sandboxes. A sandbox wrote a file and called `fsync()`, the system call a program makes to ask that its writes be stored permanently. As soon as `fsync()` returned, we took a snapshot of the Drive and opened it in a second sandbox. The file wasn’t there in any of 21 trials. Forks of the Drive taken at the same moment were missing it too.

**A Vercel Drive after fsync returns**

A sandbox wrote a 4 KiB file and a 64 MiB file, called `fsync()` on each, and synced the directory. After a delay, a new sandbox read the Drive through a snapshot or a fork. Taken the moment `fsync()` returned, 10 of 10 snapshots, 11 of 11 snapshots taken while the writer kept writing, and 11 of 11 forks were missing the file. At 1 second, every snapshot and fork still lacked it. At 30 seconds, 7 of 10 snapshots and 9 of 10 forks lacked it, and 4 of 10 snapshots taken while the writer kept writing held a 64 MiB file whose last 12 to 28 MiB read as zeros. From 60 seconds on, the file was intact in every trial but one. An Archil disk mounted in a new sandbox the moment `fsync()` returned had the file in 20 of 20 trials.

A snapshot isn’t a crash. But a snapshot is read from the storage that the Drive would be rebuilt from if its host failed, and for most of the first minute that storage didn’t have the write. Thirty seconds after `fsync()` returned, some snapshots held the file with its last megabytes reading as zeros. Only from about a minute on did snapshots and forks reliably hold it. (The script and every result are in [Vercel Drive fsync durability](https://docs.archil.com/reference/benchmarks/vercel-drive-durability). Vercel describes Drives as having “NVMe speed writes” in its [Drives documentation](https://vercel.com/docs/sandbox/concepts/drives).)

If the host had died during that minute, the write would have died with it, after the application had already been told it was safe. That matters most for the software that relies on `fsync()` most. When a database tells you a transaction committed, it has called `fsync()` and waited for it to return. A user is told their account exists, or a traveler that their seat is booked. Lose the write after that, and the account is gone when they log in and the seat goes to someone else.

## Machines that don’t come back

The Linux manual page for `fsync()` promises that once it returns, “all changed information can be retrieved even if the system crashes or is rebooted.” The guarantee was written for a machine with a disk inside it, where the worst that usually happens is a restart.

In the cloud, the machine itself is often what fails. When AWS detects an “irreparable failure of the underlying host” for an EC2 instance, it schedules the instance to stop or terminate, and data on the host’s own drives doesn’t survive a stop or a termination. (From the AWS pages on [instances scheduled to retire](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/schedevents_actions_retire.html) and [instance store data persistence](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-store-lifetime.html). The same table says instance store data survives a reboot and a power failure, which is the case `fsync()` was designed for.) A sandbox runs on whichever host has room, so its local disk is that host’s disk, and it goes wherever that host goes.

Try it in the following figure. Pick what fails, then drag the moment of the crash.

**Pull the plug**

A program writes `v1`, writes `v2`, calls `fsync()`, and then writes `v3`. Each band shows what the program is guaranteed to read back if the crash happens at that moment, for three kinds of failure. After `fsync()` returns, every band should show `v2` or later.

When only the program crashes, every kind of storage keeps every write, because the operating system still holds them. When the power is cut, every kind keeps the writes that `fsync()` covered, because the machine restarts with its disks. When the machine is lost, only storage that replicated the data to other machines before `fsync()` returned still has `v2`. The local disk loses everything, and the disk that uploads later loses `v2` until the upload finishes.

A test that kills your program and then checks the file proves nothing about storage, because the operating system still has the write. A test that cuts the power passes on almost anything. The failure that separates storage products is the one you can’t easily stage: the machine never comes back. For how a write travels from `write()` down to the drive and out to other machines, see [Where fsync returns from](https://archil.com/guides/how-file-systems-work/where-fsync-returns-from).

## Two kinds of persistent disk

Many sandbox platforms now market a persistent disk for agents, and they don’t all persist data the same way. There are two models, and the difference lies entirely in what happens when your program calls `fsync()`.

**Two ways to acknowledge fsync**

In the first model, the local drive acknowledges `fsync()` and an upload follows. The machine is lost before the upload finishes, and the write that the application was told was safe never arrives. In the second model, the write is replicated across availability zones before `fsync()` returns. Losing the machine loses nothing, and the data reaches the bucket later.

The first model acknowledges `fsync()` once the data reaches the drive inside the machine and copies it to object storage in the background. Vercel Drives behave this way in our test. Fly.io’s documentation says a Sprite’s data “is written to fast NVMe storage” while it runs and is “backed up to durable object storage” when it goes idle. (From the [Sprites overview](https://docs.fly.io/sprites/).) Both companies are making a reasonable trade for speed. Benchmarks get faster, and a database’s write speed is its `fsync()` speed. But until the copy finishes, the write exists on one drive in one machine.

The second model replicates the data to other machines before it acknowledges, so no single machine can take the write with it. EBS replicates every volume within its availability zone, and S3 stores objects across at least three zones. Local NVMe can be safe too, with a replication layer above it: PlanetScale Metal runs databases on local drives and makes sure “every write has reached stable storage in two availability zones before it’s acknowledged to the client.” (From the AWS pages on [EBS features](https://docs.aws.amazon.com/ebs/latest/userguide/EBSFeatures.html) and [S3 data protection](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html), and the FAQ on [PlanetScale Metal](https://planetscale.com/metal).)

Archil uses the second model. Our client keeps your writes in memory, like a local file system, and when you call `fsync()` it sends them to our servers, which replicate them across availability zones before they acknowledge. Archil is designed for 99.999% durability for recent writes before they reach S3. Your bucket receives them later, usually tens of seconds to minutes afterward.

That safety costs time on every `fsync()`. A synchronous 4 KiB write on Archil takes about 2 ms at the median. A program that waits for each commit before starting the next can make at most about 500 commits per second at that speed (1 second divided by 2 ms), which is fine for an agent’s workspace and slow for a write-heavy database. (Archil’s figures, with their method, are on the [performance page](https://docs.archil.com/details/performance).) Writes your program hasn’t synced are still only in our client’s memory, and a power cut loses them, as it would on a local disk. Unmount with `archil unmount`, which flushes, rather than `umount`, which doesn’t.

## A timing test you can run

You can’t see where a write goes, but you can time it. Storing data on a second machine takes at least one network round trip and a write on the far side. A drive inside the machine needs neither. Run this on the storage you’re evaluating:

```bash
# About 1,000 small writes, each followed by fdatasync().
fio --name=sync --directory=/mnt/test --rw=write --bs=4k --size=4m \
    --fdatasync=1 --ioengine=sync
```

fio prints percentiles for the `sync` latency. Compare them with our own measurements of a similar operation in the following figure.

**Where fsync answers from, timed**

The median time to create a file, write 4 KiB, call `fsync()`, and close it, on a log scale. Two acknowledge from a drive in the same machine: a Vercel Drive took 0.18 ms and a Fly.io Sprite’s disk took 3.5 ms. Two store the write on other machines first: Archil took 4.2 ms from an EC2 instance in the same region, and Mountpoint for Amazon S3, which uploads the object before `fsync()` returns, took 244 ms.

The test only works in one direction. A median of a fraction of a millisecond, like the Drive’s, most likely came from a drive in the machine, although replicated block storage can come close: AWS designs io2 Block Express volumes, which replicate within their zone, for average latency under 500 microseconds. (From the AWS page on [EBS volume types](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volume-types.html).) A slower answer proves less. The Sprite’s disk takes almost as long as Archil and still acknowledges locally.

So use the timing to decide what to ask, and ask in words a vendor can’t sidestep. When `fsync()` returns, which machines hold the data, and is the one running my code among them? If that machine dies a millisecond later, what does the next reader see?