Is it really a file system?

Run an agent’s usual tools on storage that only looks like a file system, and the first error usually comes from a tool, not from the storage. On Mountpoint for Amazon S3, git init can’t finish: git writes .git/config to a lock file and renames it into place, and Mountpoint refuses renames on general purpose buckets. On storage without symbolic links, npm install fails. A script that was executable in the repository checks out without its executable bit, and ./build.sh answers Permission denied.

Other failures make no noise. SQLite changes one 4 KiB page, and an s3fs mount uploads the whole database file again. pnpm, which normally hard-links each package from a shared store, finds no hard links and copies every file instead. Nothing reports an error, and every loop the agent runs gets slower.

These tools were written against POSIX, the interface that Unix file systems share, and agents run them because they were trained on them. This page covers the five calls behind those failures, a script that tests for them, and where Archil falls short. For the model underneath, from inodes to how an object store maps onto them, see From a path to an inode.

What each tool calls§

git: a lock file and a rename

Git changes its own files the same way every time. To update the index, a branch, or .git/config, it creates <name>.lock with O_CREAT|O_EXCL, writes the new contents into it, and renames the lock over the original. The exclusive create keeps a second git process out, and the rename publishes the change. Git’s source says that readers “are guaranteed to see either the old contents of the file or the new contents”, provided the file system implements rename(2) atomically.From git’s lockfile.h. Writes to .git/config go through the same lock file, in config.c.

Storage without rename() stops git outright, as in the Mountpoint case at the top of this page. Mountpoint’s README says not to work on a Git repository at all.From the Mountpoint README and its semantics document. Mountpoint can rename single files on S3 Express One Zone. s3fs fakes a rename with a server-side copy and a delete, so git runs, but its README lists atomic renames among the things it can’t provide.From the s3fs README.

SQLite: writes in place

In its default rollback-journal mode, SQLite copies the pages it’s about to change into a journal file, then writes the changes into the database file at those pages’ offsets.See SQLite’s Atomic Commit, sections 3.5 and 3.9. A transaction that changes one row writes a few KiB into a file that may hold gigabytes.

An object can’t be changed in place. Mountpoint accepts writes only in order from the start of a file, so SQLite can’t update an existing database on it at all. s3fs accepts the write and rewrites the whole object when it flushes, and SQLite flushes on every commit. For a 1 GiB database, that’s 262,144 times the data that changed (1 GiB ÷ 4 KiB), on every commit.

npm: symbolic links in .bin

npm installs each package’s commands into node_modules/.bin as symbolic links, and that’s how npm test finds jest and npm run build finds tsc.See npm’s folders documentation. The links are made in bin-links, which passes a failed symlink() up as an error. On storage that refuses symlink(), the install fails. npm’s bin-links setting can turn the links off, which its documentation suggests because “some file systems don’t support symlinks”. Then .bin stays empty, and scripts can’t find the commands they call.

Git meets the same gap more quietly. On storage without symbolic links, a clone checks each link out as a small plain file that contains the link’s target, and the build that follows the link is what fails.This is git’s core.symlinks setting. See git-config.

pnpm: hard links from a store

pnpm keeps one copy of each package file in a content-addressable store and hard-links it into every project’s node_modules, so a hundred projects that use a package share one copy on disk. Its default node_modules layout is built from symbolic links as well.See pnpm’s motivation and the packageImportMethod setting. When it can’t hard-link or clone, pnpm falls back to copying. The install succeeds, writes every byte of every package, and takes that much space again in each project.

The executable bit

Git records one permission bit per file: whether it’s executable. When git init or git clone starts, it flips that bit on .git/config and reads it back to see whether the file system keeps it. If the bit doesn’t stick, git sets core.fileMode to false and stops tracking it.The probe is in git’s setup.c. Git’s index stores only mode 0755 or 0644 for a regular file. Mountpoint, for example, assigns modes at mount time and chmod can’t change them. On storage like that, git status stays clean, and the script that should run doesn’t.

Lookalikes and real file systems§

Two kinds of network storage pass all five tests, because both run a real file system. A network block volume, such as EBS or Persistent Disk, runs ext4 or XFS in your own kernel. NFS, as in Amazon EFS or Filestore, sends each call to a server whose file system carries it out, and its protocol says that a RENAME “must be atomic to the client”.From RFC 1813, section 3.3.14. Their limits lie elsewhere: a block volume serves one machine at a time, and NFS waits a round trip for each create and rename. For more information, see How fast is it on small files? and Can many machines use it at once?

Object storage over FUSE fails because the bucket underneath has no rename, no partial writes, and no links. A FUSE client can refuse those calls, as Mountpoint does, or fake them with copies, as s3fs does. Neither can add an operation that the store doesn’t have.

Whole-file APIs fail more completely. Some agent toolkits give the model readFile and writeFile calls over a database or a bucket. Git, npm, and SQLite call the kernel, not the toolkit, so they don’t run there at all. Only the commands that the toolkit reimplements do.

Check it yourself§

The following script tests each call in a few seconds. Run it on the storage you’re evaluating, with the path of a directory there:

#!/bin/sh
# Usage: sh posix-check.sh /path/on/the/storage/you/are/testing
dir=${1:?usage: sh posix-check.sh DIR}/posix-check.$$
mkdir "$dir" && cd "$dir" || exit 1
check() { if eval "$2"; then echo "ok    $1"; else echo "FAIL  $1"; fi; }
 
# Rename over an existing file 200 times while a reader watches for a gap.
echo v0 > target
( i=1; while [ $i -le 200 ]; do echo "v$i" > tmp && mv -f tmp target || exit 1; i=$((i+1)); done ) &
gaps=0
while kill -0 $! 2>/dev/null; do [ -s target ] || gaps=$((gaps+1)); done
check "rename over an existing file" 'wait $! && [ $gaps -eq 0 ]'
 
mkdir -p a/b && echo x > a/b/f
check "rename a directory" 'mv a z && [ -f z/b/f ] && [ ! -e a ]'
 
# Overwrite 6 bytes in the middle of a 1 MiB file.
dd if=/dev/zero of=big bs=1024 count=1024 2>/dev/null
printf 'abcdef' | dd of=big bs=1 seek=524288 conv=notrunc 2>/dev/null
check "write at an offset" '[ $(wc -c < big) -eq 1048576 ] && [ "$(dd if=big bs=1 skip=524288 count=6 2>/dev/null)" = abcdef ]'
 
check "symbolic link" 'ln -s big sym && [ "$(readlink sym)" = big ] && [ -f sym ]'
 
# A hard link is a second name for the same file, not a copy.
ln big twin 2>/dev/null; printf 'Z' | dd of=twin bs=1 conv=notrunc 2>/dev/null
check "hard link" '[ "$(ls -l big | awk "{print \$2}")" = 2 ] && [ "$(dd if=big bs=1 count=1 2>/dev/null)" = Z ]'
 
printf '#!/bin/sh\necho ran\n' > run.sh
check "executable bit" 'chmod +x run.sh && [ "$(./run.sh 2>/dev/null)" = ran ]'
 
check "fsync" 'python3 -c "import os,sys; fd=os.open(sys.argv[1], os.O_WRONLY); os.write(fd, b\"y\"); os.fsync(fd)" big'
 
cd .. && rm -rf "$dir"

Every line should say ok. The script has three blind spots. It runs on one machine, so it can’t show what a second client sees during a rename. Run the reader loop on another machine for that. It checks that fsync() returns, not that the data survives a lost host. For that test, see When is a write safe? And it checks behavior, not cost: s3fs passes the offset write while it uploads the whole file.

For a broader test, pjdfstest is a public suite that exercises POSIX system calls one by one, including chmod, link, rename, symlink, and unlink. It runs as root from a directory on the file system under test.

Where Archil falls short§

Archil is a POSIX file system that mounts on Linux through FUSE. It supports random writes and appends, atomic renames of files and whole directories, symbolic and hard links, modes and ownership, and files that stay open after they’re unlinked. Git, SQLite, PostgreSQL, and npm run on it unchanged.The full list is in Application compatibility. It has two gaps:

• No POSIX ACLs. Mode bits and ownership work. Access control lists don’t.
• Some rename flags. A renameat2 call with RENAME_NOREPLACE, RENAME_EXCHANGE, or RENAME_WHITEOUT fails with EINVAL, so an Archil disk can’t be an overlayfs upper directory.A persistent sandbox’s root volume is ext4 and supports these flags, overlayfs, and Docker’s overlay2. The gaps apply to disks mounted through FUSE.

A disk synced to your own bucket adds a translation that mounted clients don’t see. The clients get real links, but the bucket stores a symbolic link as a small text file that holds its target, and a hard link as a full copy. A pnpm store on such a disk is shared on the mount and copied in full in the bucket.See Data sources. For more information about what the bucket holds, see Where does your data live?