# 4. Can many machines use it at once?

Source: https://archil.com/guides/evaluating-storage/can-many-machines-use-it-at-once · Part of Evaluating storage for agents · Facts checked 2026-11-04

**The short answer:** Almost any storage mounts on two machines. What differs is what the second machine sees, how soon, and what happens when both save the same file, which on most shared storage is that one save silently replaces the other. On Archil, many clients can write the same disk and the same files, and other clients see a change when their cached copy expires or when they ask for it.

Two agents share a repository. Each one reads `plan.md`, changes a different line, and saves the file. Both saves succeed, and one agent’s change is gone. A reviewer agent on a third machine runs the tests right after the writer commits, and it tests the code from before the commit. Move the repository to a network block volume, and the second machine can’t mount it at all.

Not one of these raises an error. Each comes from a choice the storage makes about sharing, and vendors choose differently. This chapter shows each failure, then what the three common ways agents share files need from storage.

## Lost updates

**Two saves, one change lost**

Agents A and B each read `plan.md`, which has three lines. A changes line 1 and B changes line 3. A saves first. Then B saves its whole copy, which still has the old line 1. Both saves return success. The file ends up with B’s new line 3 and the old line 1, so A’s change is lost, and neither agent sees an error.

An agent saves a file by writing out its own copy, and that copy came from a read made before the other agent’s edit. Storage can’t merge the two, because it never sees edits, only bytes. Shared sandbox volumes call this rule “last write wins”: Modal’s documentation uses those words, and Daytona documents its Volumes the same way. NFS behaves the same, unless every program takes a lock first.

What storage can do is refuse the second writer. Then the agent gets an error, reads the file again, and reapplies its edit. Two saves at the same moment can also do worse than lose an edit: their writes can interleave and leave a file that matches neither version. [How it works](https://archil.com/guides/how-file-systems-work/one-file-many-caches#writes-are-bytes-at-offsets) shows why.

## Stale reads

We created a file on one machine and timed how long a second machine took to see it, on six setups.

**Until another machine sees a new file**

We created a file on one client and measured how long a second client took to see it. Archil, when the reader runs `archil invalidate-cache` first: about 70 ms. Mountpoint for Amazon S3: 284 ms, with no metadata cache by default. A Daytona Volume: about 360 ms, the median of 10 runs. Amazon S3 Files: 30 s, the length of its NFS directory attribute cache. Archil with default client caching: 30 s, its client cache expiry. A Modal Volume: not within 60 s, because a running sandbox sees another sandbox’s commit only after it reloads the Volume.

The slow results aren’t slow networks. Every client caches what it has looked up, so it can answer `stat` and `open` without asking a server. It keeps each entry for a set time and answers from memory until that time runs out. Amazon S3 Files mounts over NFS, and the Linux NFS client keeps a directory’s attributes for at least 30 s by default. (The `acdirmin` default, from the [nfs(5) man page](https://man7.org/linux/man-pages/man5/nfs.5.html).) Archil’s client keeps entries that long by default as well. Mountpoint keeps no metadata cache by default, so it asks S3 every time, at the cost of a round trip per lookup. A Modal sandbox sees another sandbox’s commit only after it reloads the Volume, which works only when no files are open.

For an agent, the window decides what a reader can trust. A reviewer that looks too soon tests old code. A coordinator that polls for `done.json` waits out the cache before it sees the file. Ask how a reader gets the current state on demand. On Archil, the reader runs `archil invalidate-cache` on the mount before it reads. [How it works](https://archil.com/guides/how-file-systems-work/one-file-many-caches) covers how caches learn that they’re stale.

## Volumes that attach once

A network block volume avoids every problem so far by refusing to share: an EBS gp3 volume attaches to one instance at a time. That isn’t an arbitrary limit. The file system on the volume, such as ext4 or XFS, keeps its map of which blocks belong to which files in one kernel’s memory, and a second kernel writing the same device would overwrite that map. (EBS Multi-Attach lets one `io1` or `io2` volume attach to up to 16 Nitro-based instances in one availability zone. AWS’s documentation says standard file systems such as XFS and EXT4 “are not designed to be accessed simultaneously by multiple servers”. For more information, see [Multi-Attach](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes-multi.html).)

**One volume, one machine**

Machine B asks to attach the volume while machine A holds it, and the attach is refused. Only after A unmounts and detaches the volume can B attach it. The volume never belongs to two machines at once.

A handoff is a sequence: unmount, detach, attach, mount. That works for an agent that resumes later on another machine. A reviewer that wants to read while the writer works can’t do it at all.

## Three ways agents share

Agent systems share files in three common patterns, and each needs something different from storage.

**Three ways agents share files**

Three patterns. In a parallel fan-out, five attempts at one task each work on their own branch of the workspace, and the one that works is kept. Beside a writer, one agent writes and several reviewers read the same files. In a handoff, a sandbox stops partway through a task, and a new sandbox mounts the same files and continues from where it stopped.

- **A parallel fan-out** runs several attempts at one task. The attempts don’t need to share a writable directory, and sharing one invites every failure in this chapter. They need isolation: a copy per attempt, so you can keep the one that worked. On Archil-managed disks, a checkpoint and a branch take 13 ms and 33 ms at the median, whatever the disk’s size. For more information, see [Can you fork it?](https://archil.com/guides/evaluating-storage/can-you-fork-it)
- **A reviewer beside a writer** is one writer and many readers. The writer needs nobody else writing its files. The readers need a known limit on how stale a read can be, and a way to get the current state on demand.
- **A handoff** moves a task from one sandbox to the next, often because the first one timed out. The new sandbox needs every write the old one covered with `fsync()`. It also needs the old sandbox’s write access to end, including when that sandbox died without releasing it, at the cost of no more than the writes it never sent.

## Archil’s mount modes

Any number of Archil clients can write one disk, and you choose how they coordinate. In conditional mode, several clients write the same paths at once, and each write runs on our servers, as it would over NFS. When one client does most of the writing in a path, it can check that path out instead. While it holds the checkout, other clients read but don’t write there. (Our docs call a checkout a delegation. For more information, see [Sharing disks](https://docs.archil.com/concepts/sharing-disks).) The holder answers `open`, `write`, and `rename` from memory, like a local disk, and sends its changes to our servers about once a second and on every `fsync()`.

How much of this you manage depends on the mode you mount in:

- **Exclusive mode**, the default, checks out the whole disk for one client. Other clients can mount it with `--read-only`.
- **Shared mode** (`--shared`) leaves the disk read-only on each client until it runs `archil checkout <path>`. Opening a file for writing, or creating one, checks out that file automatically. Deleting or renaming needs the parent directory checked out.
- **Conditional mode** (`--conditional`) lets a client write anywhere that nobody has checked out. Creates, renames, and deletes run on our servers before they return, and the last write to reach the disk wins. Its performance is that of a remote file system.

**What client B can do, by mode**

Client A has checked out `/repo` (in exclusive mode, the whole disk). In exclusive mode, B mounts with `--read-only`: it can read everything, and every change fails as read-only until A unmounts. In shared mode, B can read, can’t change `/repo/plan.md` until A checks it in, can write a file nobody holds (opening it checks it out until B closes it), can’t delete a file until it checks out the parent directory, and writes at local speed after `archil checkout /data`. In conditional mode, B can read, changes and deletes outside `/repo` run on the server at remote-file-system speed with the last write winning, and a write to `/repo/plan.md` can return and then fail when it’s sent: B’s mount turns read-only and its unsent writes are lost.

The modes don’t mix freely. If another client checks out a path that a conditional client is writing to, or mounts the disk in exclusive mode, the conditional client’s mount turns read-only, and the writes it hadn’t sent are lost. (For more information, see [Writes fail on a conditional mount](https://docs.archil.com/reference/troubleshooting#problem-writes-fail-on-a-conditional-mount).)

For the three patterns:

- **Fan-out:** give each attempt its own branch, mounted exclusive.
- **Reviewer:** mount the writer exclusive and the reviewers `--read-only`. Have each reviewer run `archil invalidate-cache` before it reads.
- **Handoff:** the new sandbox mounts once the old one has unmounted. If the old one died while holding the disk, `archil mount --force` or `archil checkout <path> --force` takes over. That loses whatever the old client hadn’t sent, the way force-detaching an EBS volume does.

## Two tests to run on any shared storage

You can test both failures in a few minutes with two machines that mount the same storage at `/mnt/shared`. First, visibility. Start the loop on machine B, then create the file on machine A, and compare the two timestamps:

```bash
# Machine B: cache the directory, then wait for the file to appear.
ls /mnt/shared > /dev/null
until [ -e /mnt/shared/probe ]; do sleep 0.05; done; date +%s.%N

# Machine A:
touch /mnt/shared/probe; date +%s.%N
```

The difference, with both clocks synced by NTP, is how long a reader can be stale. Next, lost updates. Run this on both machines at the same moment, then count the lines:

```bash
for i in $(seq 1000); do echo "$(hostname) $i" >> /mnt/shared/log; done
wc -l /mnt/shared/log
```

Fewer than 2,000 lines with no error on either machine means the storage lost writes silently. On NFS, `O_APPEND` is simulated by the client, and the Linux `open(2)` man page warns that it “may lead to corrupted files” when several machines append at once.

## Where Archil falls short

Readers don’t see another client’s changes until their cached entries expire, unless they run `archil invalidate-cache`. Checkouts are managed by hand today: your code decides who checks out what, and when to check it in. Conditional mode lets many clients write the same files, but at the speed of a remote file system, and a conditional client loses its unsent writes if another client checks out the path it’s writing to.