How fast is it on small files?

Installing the dependencies of one Next.js app with npm ci writes 18,602 files. We ran that install on one EC2 instance with several file systems mounted side by side, and stopped the clock only when every write was durable. On the instance’s own EBS volume, it took 10.7 s. On Archil, it took 12.4 s. On Amazon S3 Files, which serves an S3 bucket over NFS, the same install took 71.4 s.

The host, the package cache, and the bytes written were the same each time. What differed was how often each file had to wait for a server. An agent that clones a repository, installs its dependencies, and runs its tests pays that difference at every step, and wall-clock time is what you and your users notice.

Two jobs, five file systems§

The pattern holds for git clone and for other designs. On the same host, a clone of the Django repository took 15.5 s on local disk, 19.8 s on Archil, and 107.5 s on S3 Files. JuiceFS, which keeps its metadata in a database (here Redis on ElastiCache, in the same Availability Zone), took 161.6 s for the install and 247.4 s for the clone with its defaults. Its write-back mode stages writes on local disk and uploads them later. Timed until the upload finished, it took 53.8 s and 29.2 s.

A separate benchmark against Amazon EFS shows the same shape. Cloning React (7,260 files) took 9.8 s on Archil and 81.5 s on EFS. npm ci of a create-next-app project took 7.6 s on Archil and 86.3 s on EFS. That benchmark stopped the clock when each command returned, rather than when its writes were durable.For the hosts, versions, and commands behind both benchmarks, see File systems on one host.

Where the time goes§

Divide each install by its 18,602 files. On local disk, 10.7 s is 0.58 ms per file. That’s npm’s own work: reading its cache, unpacking, and making system calls that the kernel answers from memory. On S3 Files, 71.4 s is 3.84 ms per file, so each file spent about 3.3 ms waiting on the network. On Archil, each file spent about 0.09 ms.

A round trip to a server in the same region is short. Our published 4 KiB read that misses the client’s cache is one round trip, and it takes 0.50 ms at the median. So each file on S3 Files waited for the equivalent of six or seven round trips.

The waits come from how a network file system creates a file. A program makes four calls: open with O_CREAT, write, fchmod, and close. A local file system answers all four in memory. An NFS client has to send three of them to the server, as CREATE, SETATTR, and WRITE followed by COMMIT, and wait for each answer.

The client can’t skip the question, even when no one else is using the directory. If two machines both create hello.txt, only the server sees both requests, so only the server can decide which one gets the name. The client has no way to know that nobody else is looking, so it asks every time.

A client can answer from memory only once it holds the right to decide: the server has handed it the directory, and nobody else can change that directory until the client hands it back. We call that a checkout. NFS and our docs call it a delegation. By default, an Archil client checks out the whole disk when it mounts it. It answers open, write, chmod, and close from memory, and it sends the changes to our servers later, many files to a message, and whenever the program calls fsync(). For how this idea grew out of AFS callbacks, leases, and NFSv4 delegations, see Anatomy of a file create over a network.

A test you can run§

Run the job you care about on the same host, once on local disk and once on each storage you’re considering. Stop the clock when the writes are durable, not when the command returns. A design that stages writes on a local disk and uploads them later looks like local disk until you wait for the upload.

# Set MNT to each mount in turn, local disk first.
time (git clone -q file:///data/repo.git "$MNT/repo" && sync -f "$MNT/repo")
time (cd "$MNT/app" && npm ci --prefer-offline --ignore-scripts && sync -f .)

sync -f flushes the whole file system that holds the path. On Archil, run archil sync <mountpoint> instead. Keep the inputs on local disk, as in the preceding commands, so the network to GitHub or the npm registry isn’t part of the time. Then divide by the number of files. The following calculator starts from our install and takes your numbers too.

Where Archil is slower§

Small files aren’t the only workload, and on several others we lose. In the same EFS benchmark, EFS was faster on the following three tests.

We have more work to do on throughput.

A checkout makes one client fast by making the others ask first. In the default exclusive mode, other machines can mount the disk read-only, but they can’t write until the holder unmounts. To share writes, clients check out separate paths, or mount in conditional mode, where creates and renames run on the server as they do on NFS, at the speed of a remote file system. When many clients keep writing the same files, handing ownership back and forth can cost more than NFS’s round trips. Other clients see changes within about 30 seconds by default, or at once after archil invalidate-cache. For more information about sharing, see Can many machines use it at once?

Distance still counts for reads. A checkout removes the waits on files a client writes, but a file the client has never read has to cross the network. A read that also misses our servers’ cache goes to the object store, typically 10 to 30 ms on S3. From a Modal sandbox about 40 ms from our servers, Archil read a 1 GiB file into a new sandbox at 331 MB/s, and a Modal Volume read it at 624 MB/s. Keep compute in the same region as the disk, warm the cache with archil prefetch before a job reads a tree it hasn’t seen, or run the command next to the data with serverless execution.