# 2. Where fsync returns from

Source: https://archil.com/guides/how-file-systems-work/where-fsync-returns-from · Part of How file systems work · Facts checked 2026-11-04

**In this chapter:** How one write travels from a program’s memory to a drive’s flash and then to other machines, what `write()`, `fsync()`, `fdatasync()`, and `O_DIRECT` each wait for, and why replicated storage acknowledges where it does. By the end, you can name the layer any storage system acknowledges from, and the failures that choice survives.

Between your program and permanent storage sit several places a write can rest: the program’s own memory, the kernel’s memory, a queue on its way to the drive, the drive’s cache, the flash itself, and then other machines. Each layer above the flash exists because the one below it is slower, and each can tell the layer above that a write is done.

Durability is the question of which layer a call waits for. This page follows a single 4 KiB write down through one Linux machine, then out across several. The examples are Linux and ext4, because their behavior is documented in detail, but the same layers appear in every operating system and file system.

## The page cache

A drive is slow next to memory, and programs write in small, scattered pieces. If every `write()` went to the drive before returning, a program writing a log line at a time would wait on the drive for every line. So the kernel keeps file data in memory, in the page cache, and serves both reads and writes from there.

When your program calls `write()`, the kernel copies the bytes into a page of that cache, marks the page dirty, and returns. Dirty means the page holds data the drive doesn’t have yet. The cache is shared by every process on the machine, so a program that reads the file a moment later gets the new bytes, even though the drive has never seen them.

**Writes land in memory**

A program calls `write()` twice. Each write lands in the page cache, which marks the page dirty, and `write()` returns. Then the program calls `fsync()`: both writes go down to the disk, the pages become clean, and `fsync()` returns once the data is there.

This explains a common mistake in durability testing. If you kill a process and then read its file, you read the page cache, which still has every write. Every kind of storage passes that test, so it tells you nothing.

## Writeback on the kernel’s schedule

Dirty pages don’t stay in memory forever. Kernel flusher threads wake up periodically and write out pages that have been dirty for long enough. In the Linux source, the defaults are a wake-up every 5 seconds and an age of 30 seconds. (`dirty_writeback_interval = 5 * 100` and `dirty_expire_interval = 30 * 100`, both in centiseconds, in the kernel’s [mm/page-writeback.c](https://github.com/torvalds/linux/blob/master/mm/page-writeback.c). They’re tunable as `vm.dirty_writeback_centisecs` and `vm.dirty_expire_centisecs`, described in the [kernel’s vm sysctl documentation](https://docs.kernel.org/admin-guide/sysctl/vm.html).)

**A write nobody syncs**

A program dirties a page at 2 seconds and never calls `fsync()`. Flusher threads wake every 5 seconds and write pages that have been dirty for 30 seconds, so the page is written at the 35-second wake-up. A power cut at any time in between loses the write, although `write()` returned long before.

Memory pressure also forces writeback. When dirty pages pass a share of available memory, `dirty_background_ratio`, the flusher threads start writing in the background. Past a second, larger share, `dirty_ratio`, a process that keeps writing has to write out dirty data itself, so its `write()` calls slow down. The kernel source sets these at 10% and 20%.

Most of the time, this is enough. A laptop that crashes can lose up to about half a minute of unsynced writes, and the programs that care call `fsync()`. Your Mac asks you to eject an SD card for the same reason: ejecting flushes what’s still in memory before you pull the card.

## The calls that wait

A program that needs a write to survive a power cut has to ask. The Linux manual page for `fsync()` says it flushes a file’s modified data “to the disk device (or other permanent storage device) so that all changed information can be retrieved even if the system crashes or is rebooted.” Two more sentences matter as much. The flush “includes writing through or flushing a disk cache if present,” and the call “blocks until the device reports that the transfer has completed.” (From the [fsync(2) manual page](https://man7.org/linux/man-pages/man2/fsync.2.html).)

`fsync()` flushes the file’s metadata too: its size, its timestamps, and where its blocks are. `fdatasync()` skips metadata that a later read doesn’t need. The man page’s example is the modification time, which a read can do without, whereas a change in file size has to be flushed. On Linux, `fdatasync()` is PostgreSQL’s default way of syncing its log. (The `wal_sync_method` setting in [PostgreSQL’s write-ahead log settings](https://www.postgresql.org/docs/current/runtime-config-wal.html).)

You can also make every write wait, by opening the file with a flag. With `O_SYNC`, each `write()` behaves as if `fsync()` followed it. With `O_DSYNC`, it behaves as if `fdatasync()` followed it.

`O_DIRECT` is different, and often misread. It tells the kernel to skip the page cache and move data between the program’s memory and the drive directly. Databases that manage their own cache use it to avoid holding every page twice. But skipping the cache isn’t a durability promise. The `open()` man page says `O_DIRECT` alone only “makes an effort to transfer data synchronously,” and that you need `O_SYNC` with it to get the full guarantee. (From the [open(2) manual page](https://man7.org/linux/man-pages/man2/open.2.html).) An `O_DIRECT` write can return while the data sits in the drive’s own cache.

Step a write through the stack in the following figure. Each choice shows where the call returns, and what a crash, a power cut, or a lost machine would leave behind at that moment.

**One write, step by step**

The figure steps a 4 KiB write, and the file’s modification time, through five layers of one machine: program memory, the page cache, the queue to the drive, the drive’s write cache, and flash. A plain `write()` returns once the data is in the page cache. The page reaches the drive about 30 seconds later, and flash whenever the drive gets to it. `fdatasync()` returns only after the kernel has sent the data and a cache flush, so the data is on flash, but the modification time stays in memory. `fsync()` does the same for the data and the modification time. An `O_DIRECT` write skips the page cache and returns once the drive reports completion, which can be while the data is still in the drive’s volatile cache. `O_DIRECT` with `O_DSYNC` returns once the data is on flash. Data in the page cache survives a program crash. Only data on flash, or in the cache of a drive with power-loss protection, survives a power cut. Nothing in this machine survives losing the machine.

## Metadata and the journal

Creating a file changes more than its data. The file system allocates blocks for the data, writes an inode that records the file’s size and where its blocks are, and adds an entry to the parent directory that maps the name to the inode. Each lives in a different place on the drive. For more information about inodes and directory entries, see [From a path to an inode](https://archil.com/guides/how-file-systems-work/from-a-path-to-an-inode).

The `fsync()` man page warns about this: syncing a file “does not necessarily ensure that the entry in the directory containing the file has also reached disk.” For that, you also call `fsync()` on the directory.

The order of these writes matters as much as whether they happen. Suppose a crash lands after the inode is written but before the data. The file now claims blocks that hold whatever they held before, possibly another file’s old contents. Journaling exists to rule out that state. A journal is a small area of the drive written in order. The file system records a group of related changes there as a transaction and seals it with a commit record. After a crash, it replays sealed transactions and ignores unsealed ones, so each group of changes lands whole or not at all.

ext4 offers three modes, and the default, ordered mode, journals only metadata. Its documentation says ordered mode groups the metadata for a change with its data blocks, and “when it’s time to write the new metadata out to disk, the associated data blocks are written first.” ext4 commits a transaction at least every 5 seconds by default. (From the [ext4 administrator’s guide](https://docs.kernel.org/admin-guide/ext4.html), which also describes `data=journal` and `data=writeback`.)

**Ordered mode**

In ext4’s ordered mode, the new data is written to its final place on disk first. Then the metadata, meaning the inode and the directory entry, is written to the journal. A commit record seals the transaction, and from then on, recovery after a crash applies it. Later, the metadata is copied to its home in the main area and the journal space is reused.

A journal keeps the file system consistent. It doesn’t make your writes durable. That still takes `fsync()`, and programs that skip it learn the difference eventually. In 2009, users of the new ext4 found files full of zeros, or empty, after a crash. ext4 delays choosing blocks for new data, so a rename could commit before the data it pointed at had anywhere to go. Ted Ts’o, the ext4 maintainer, answered that “if you really care about making sure something is on disk, you have to use fsync or fdatasync.” ext4 still added a workaround for the common case. (From [“ext4 and data loss”](https://lwn.net/Articles/322823/) by Jonathan Corbet, LWN, March 11, 2009. The workaround is the `auto_da_alloc` mount option, on by default, which the ext4 guide introduces with “Many broken applications don’t use `fsync()` when replacing existing files.”)

The safe way to replace a file’s contents takes two syncs:

```c
int fd = open("config.tmp", O_WRONLY | O_CREAT | O_TRUNC, 0644);
write(fd, buf, len);
fsync(fd);                       // the new contents are on disk
close(fd);
rename("config.tmp", "config");  // swap the name atomically
int dir = open(".", O_RDONLY | O_DIRECTORY);
fsync(dir);                      // the rename is on disk too
close(dir);
```

## The drive’s own cache

The drive is a computer too, with its own memory. Many drives accept a write into that memory and report it complete before it reaches the flash or the platter. The kernel’s documentation describes such drives as signaling “I/O completion to the operating system before data actually has hit the non-volatile storage.” (From [Explicit volatile write back cache control](https://docs.kernel.org/block/writeback_cache_control.html) in the kernel documentation.)

The kernel has two tools for this. A flush, `REQ_PREFLUSH`, guarantees that every write the drive has already completed is on non-volatile storage. Force Unit Access, `REQ_FUA`, marks one write so that the drive reports completion only “after the data has been committed to non-volatile storage.” `fsync()` uses these, which is how it writes through the drive’s cache. Turn on power-loss protection in the stack figure to see the other way out.

The arrangement depends on the drive telling the truth. SQLite’s documentation is blunt about it: “most consumer-grade mass storage devices lie about syncing,” and USB sticks are “especially pernicious liars.” (From [How To Corrupt An SQLite Database File](https://www.sqlite.org/howtocorrupt.html), section 3.1.) Some drives built for data centers take the other route: capacitors that hold enough charge, when power fails, for the drive to finish writing its cache. Samsung’s description of the feature says the capacitors give the drive time “for the data flush from the DRAM to the NAND to occur.” (From Samsung’s [power loss protection](https://download.semiconductor.samsung.com/resources/others/Samsung_SSD_845DC_05_Power_loss_protection_PLP.pdf) white paper for its 845DC data-center drives.) A drive with no volatile cache to lose needs no flush, and the kernel completes empty flush requests without sending them to the drive.

## When fsync fails

So far, `fsync()` has either succeeded or been interrupted by a crash. It can also fail, and in 2018 the PostgreSQL project discovered what Linux did then.

PostgreSQL writes data pages through the page cache and later calls `fsync()` on each file. If the call fails, the natural reaction is to retry. Craig Ringer reported to the pgsql-hackers mailing list on March 28, 2018, that this was unsafe. (The thread is [“PostgreSQL’s handling of fsync() errors is unsafe and risks data loss at least on XFS”](https://www.postgresql.org/message-id/CAMsr%2BYHh%2B5Oq4xziwwoEfhoTZgr07vdGG%2Bhu%3D1adXx59aTeaoQ%40mail.gmail.com).) When writeback fails, LWN explained, file systems typically respond by “discarding the data in the affected pages and marking them as being clean.” The error is reported once. A retried `fsync()` finds no dirty pages, has nothing to write, and returns success. (From [“PostgreSQL’s fsync() surprise”](https://lwn.net/Articles/752063/) by Jonathan Corbet, LWN, April 18, 2018.)

**A retry that succeeds**

A program writes, dirtying a page. Writeback to the disk fails, and the kernel marks the page clean anyway, although the data isn’t on disk. The next `fsync()` returns the error `EIO`. The program retries, and the second `fsync()` returns 0, meaning success, although the data never reached the disk and the page that held it can be evicted.

The kernel developers had reasons. Ts’o pointed out that the most common cause of I/O errors is a user pulling out a USB drive, and keeping the dirty pages of a vanished drive could fill memory. Linux 4.13 and later releases changed how writeback errors are tracked, so they’re harder to miss. Each error is still reported once, and the pages are still gone. (From PostgreSQL’s [Fsync Errors](https://wiki.postgresql.org/wiki/Fsync_Errors) wiki page, which lists the behavior of each kernel version.)

PostgreSQL’s answer was to stop trusting the page cache after a failure. It now stops with a `PANIC` when `fsync()` fails, then recovers by replaying its write-ahead log, which still holds every committed change. The project made the change in version 12 and back-patched it to 9.4 through 11. The episode became known as “fsyncgate.” For a storage design, it means that after a failed sync, the page cache can’t be trusted, and recovery needs a copy written somewhere else.

## The machine as a failure domain

Everything so far survives a power cut, at best. None of it survives losing the machine, because every layer, flash included, is inside it.

In the cloud, losing the machine is routine. AWS’s documentation for EC2 instance store, the drives physically attached to the host, lists what survives. A reboot and a power failure keep the data. A stop, a termination, a retirement after the host fails, or a failed disk lose it. (From [Data persistence for Amazon EC2 instance store volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-store-lifetime.html).) A program can call `fsync()` perfectly and still lose everything it wrote there.

Storage services publish durability as a probability, and the units differ. AWS designs a gp3 EBS volume for 99.8% to 99.9% durability, which it states as a 0.1% to 0.2% annual failure rate, for the whole volume. It designs S3 for 99.999999999% durability of each object over a year. (From the AWS pages on [EBS volume types](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volume-types.html) and [S3 data protection](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html).) Both get there the same way: by putting copies on several machines before saying a write is done.

## Replicate, then acknowledge

To survive the loss of a machine, a write has to exist somewhere else before anyone is told it’s safe. The simplest form is a primary that forwards each change to a standby and waits for an answer. The design question is what the standby must do before it answers.

PostgreSQL makes the choice explicit in its `synchronous_commit` setting. With `off`, a commit returns before even the primary’s log is flushed. With `local`, it waits for the primary’s own flush “but not for replication.” With `remote_write`, it also waits until the standby has “written it to their file systems,” which means the standby’s page cache. With `on`, the default, it waits until the standby has “flushed it to durable storage.” With `remote_apply`, it also waits until the standby has applied the change, so queries there see it. (Quoted from `synchronous_commit` in [PostgreSQL’s write-ahead log settings](https://www.postgresql.org/docs/current/runtime-config-wal.html). The standby levels apply when `synchronous_standby_names` names a standby.)

Each step down the ladder adds a wait, and each protects against one more failure. Where the standby runs matters as much. A standby in the same zone shares that zone’s power and network, so one event can take both machines.

**Who acknowledges**

A primary and one standby. Acknowledging from the primary’s memory survives none of the five failures shown, not even a crash of the database process. Acknowledging from the primary’s disk survives a process crash and a power cut, and loses the commit if the primary machine is lost. Acknowledging once the standby has the write in memory survives losing the primary, but if the standby is in the same zone, a zone-wide power cut wipes its memory. Acknowledging once the standby has it on disk survives all five failures with the standby in another zone, and everything except losing the whole zone with the standby in the same zone. Acknowledging after the standby applies the change costs one more step and adds visibility on the standby, not durability.

Amazon’s Dynamo paper shows a system choosing a middle rung on purpose. Its storage nodes buffer writes in memory and write them to disk in the background. To limit the risk, the coordinator chooses “one out of the N replicas to perform a ‘durable write’,” so at least one copy reaches disk before the write counts. (From [“Dynamo: Amazon’s Highly Available Key-value Store”](https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html), DeCandia and others, SOSP 2007.)

## Quorums

Waiting for every replica has a cost: the slowest replica sets the pace, and a dead one stops all writes. A quorum waits for enough replicas instead of all of them.

Raft, a consensus algorithm, commits a log entry “once the leader that created the entry has replicated it on a majority of the servers.” Its followers update their logs “on stable storage before responding,” so a reply means a copy on disk. A cluster of five servers keeps working through any two failures. (From Diego Ongaro and John Ousterhout, [“In Search of an Understandable Consensus Algorithm”](https://raft.github.io/raft.pdf), USENIX ATC 2014.) Dynamo generalizes the idea: with N replicas, waiting for W of them on a write and R on a read, “setting R and W such that R + W > N yields a quorum-like system,” because every read then overlaps at least one replica that took the latest write.

**A majority of five**

The leader appends entry 7 to its log and sends it to four followers. One follower is unreachable and one is slow. Two followers store the entry on disk and reply, so three of the five servers have it, a majority. The leader commits and answers the client. The slow follower stores the entry later.

A quorum only helps when its members fail independently. Three replicas in one rack share a power supply and a switch. Three in one building share its power and its fire. Cloud providers draw this boundary as the availability zone: S3 stores objects “across a minimum of three Availability Zones,” which are “physically separated by a meaningful distance, many kilometers, from any other.” PlanetScale Metal runs databases on local NVMe and still makes sure “every write has reached stable storage in two availability zones before it’s acknowledged.” (From [S3 data protection](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html) and the FAQ on [PlanetScale Metal](https://planetscale.com/metal).) Crossing zones costs a longer round trip on every acknowledged write. That cost buys survival of the failures a single zone can’t protect against.

## Write-ahead logs

Replication and crash recovery both get cheaper with one more idea. Instead of updating a file’s pages in place and syncing each one, a system appends a description of the change to a log, syncs the log, and acknowledges. It updates the real data structures later, in bulk.

PostgreSQL’s documentation states the rule: changes to data files “must be written only after those changes have been logged.” Because of it, only the log “needs to be flushed to disk to guarantee that a transaction is committed,” and the log “is written sequentially,” which costs less than flushing scattered pages. (From [PostgreSQL’s introduction to write-ahead logging](https://www.postgresql.org/docs/current/wal-intro.html).) After a crash, replaying the log redoes whatever hadn’t reached the data files.

**Log first, data later**

Four changes to a balance are appended to the log, and each commit is acknowledged once its record is durable. The data files still say 100 when the machine crashes. On restart, replay applies the four records, the data files say 160, and the log up to the fourth record can be trimmed.

The log is also the natural thing to replicate. PostgreSQL streams its log to standbys, and Raft is an algorithm for agreeing on a log. A storage system can keep recent writes in a replicated log and fold them into slower storage, such as an object store, long after they were acknowledged.

## The layers at a glance

The following table lists each place a write can be acknowledged from, and which failures a write acknowledged there survives.

| Acknowledged from                               | Program crash | Power cut | Machine lost | Zone lost                    |
| ----------------------------------------------- | ------------- | --------- | ------------ | ---------------------------- |
| Program memory (a buffered library call)        | No            | No        | No           | No                           |
| Page cache (`write()`)                          | Yes           | No        | No           | No                           |
| Drive’s volatile cache (`O_DIRECT`)             | Yes           | No        | No           | No                           |
| Drive’s flash, or a protected cache (`fsync()`) | Yes           | Yes       | No           | No                           |
| Another machine’s memory                        | Yes           | Yes       | Yes          | Only if it’s in another zone |
| Another machine’s disk                          | Yes           | Yes       | Yes          | Only if it’s in another zone |
| A majority of disks across zones                | Yes           | Yes       | Yes          | Yes                          |

The two rows for another machine look alike, but they differ when that machine loses power too. A write held in its memory is gone, and a write on its disk comes back.

## A worked example: Archil

Archil combines these pieces, and its design maps onto the layers on this page. Our client runs on the same machine as your program and keeps recent writes in memory, as the page cache does, so `write()` doesn’t wait for the network. The client sends changes to our servers in the background, and immediately when you call `fsync()`.

The servers append those changes to a write-ahead log and replicate it across availability zones before `fsync()` returns. In the terms of the “Who acknowledges” figure, Archil acknowledges once other machines have stored the write, and those machines are in other zones. Archil is designed for 99.999% durability for writes in that log before they reach S3. Later, usually tens of seconds to minutes after a write, the log is folded into the disk’s object storage. (From [Data durability](https://docs.archil.com/details/architecture#data-durability) and [Consistency](https://docs.archil.com/details/consistency) in Archil’s documentation.)

The page-cache rule applies to our client too. A write that hasn’t been synced exists only in the client’s memory, and a power cut loses it. Use `archil unmount`, which flushes before it detaches, rather than a plain `umount`, which doesn’t.

For what this means for agents whose machines are deleted when a task ends, see [Storage for agents](https://archil.com/guides/storage-for-agents).