How file systems workChapter 5 of 7
A file system on an object store
In this chapter
How a tree of files, with its modes, links, and renames, gets laid onto a flat map of immutable objects. By the end, you can read any product’s bucket layout and say what it gives up, what it costs to rename and list, and what happens when someone else writes to the bucket.
Hunter LeathFounder & CEO, Archil
16 min readChecked November 2026Read as Markdown
A file system and an object store look alike from a distance. Both keep bytes under names, and both survive the machine that wrote them. Up close, they answer different questions. A file system is built for programs that change a few bytes in the middle of a file, rename it into place, and ask who may read it. An object store is built for whole objects, written once and read by anyone with the key.
Every product that puts a file system in front of a bucket translates one into the other, and that translation decides what other tools find in the bucket. This page builds the translation one step at a time. It uses Linux as the file system and Amazon S3 as the object store, because Google Cloud Storage, Cloudflare R2, and many other stores accept the same API.
Two interfaces§
A Linux file system is a tree. A directory holds names, and each name points at a file: its data, its mode bits, its owner, and its size. From a path to an inode builds this model from the disk up. Because names are separate from files, one file can have two names. The link(2) man page says the new name “may be used exactly as the old one for any operation; both names refer to the same file,” and that “it is impossible to tell which name was the ‘original’.” A program can write any byte range of a file at any time. And rename(2) is atomic: when it replaces a file, there is “no point at which another process attempting to access newpath will find it missing.”From the Linux man pages for link(2) and rename(2).
An S3 bucket is a map. Each key names one object, and an object is its data plus a little metadata. There are five calls that matter here: PUT stores a whole object, GET reads any byte range of one, LIST returns keys in order, COPY duplicates an object on the server, and DELETE removes one. S3’s documentation states the rules that make this simple to reason about. “Updates to a single key are atomic,” so a reader gets “either the old data or the new data, but never partial or corrupt data.” And “there is no way to make atomic updates across keys.”From the Amazon S3 data consistency model.
Those two rules keep an object store simple to reason about. For a file system, they’re the whole difficulty. The following figure lines up what programs ask of a file system against what the object store offers.
Nine file system calls beside the five calls an object store offers. Writing and closing a file maps onto PUT. Writing into the middle of a file also maps onto PUT, but only by rewriting the whole object. A read maps onto a ranged GET, and listing a directory maps onto LIST with a prefix. A rename maps onto two calls, COPY and then DELETE. Changing a mode or owner, making a hard or symbolic link, and making an empty directory have nothing to map onto.
One object per file§
The most direct translation makes each file one object, with the file’s path as its key. The file src/app.ts becomes the object src/app.ts. Mountpoint for Amazon S3, s3fs, and Amazon S3 Files all work this way.
Directories come for free, in a sense. S3 general purpose buckets have “a flat structure instead of a hierarchy,” and what looks like a folder is a shared prefix: if the keys logs/date1.txt and logs/date2.txt exist, there is a logs folder.From AWS’s guide to organizing objects by using folders. The file system rebuilds the tree from the keys.
What this layout buys is the reason people choose it: every tool that speaks S3 reads the files where they sit. DuckDB, Spark, a training job, and aws s3 cp need nothing from the product that wrote them. What it costs is everything in the preceding figure that has no translation. The next sections go through those one at a time, and the following figure lets you try each one. Run a command, and watch both sides change and the requests each layout sends.
How the bucket is laid out
Run a command on the mount
On the mount
mode, then the name
In the bucket
each file an object at its path
Run a command to see what each layout sends to the bucket. Switch the layout at any point to compare.
So far, with this layout: 0 requests · 0 B uploaded
A workspace on the mount (app.db at 64 MiB, build.sh, notes.md, and a src directory holding three files) and what the bucket holds for it, under two layouts. Each command sends the following to the bucket.
With one object per file: chmod +x build.sh has nowhere to go unless the tool keeps modes as user-defined metadata, which S3 changes only by copying the object onto itself (one request). ln -s notes.md README.md becomes an 8-byte object, if the tool writes links as text. ln notes.md notes-v1.md becomes a second, independent 1 KiB object. mkdir out sends nothing, so the bucket has no out/ until a file is written in it. mv src code copies and deletes each of the three objects: 6 requests. Writing 4 KiB into app.db uploads the whole 64 MiB object again.
With blocks and a metadata database (JuiceFS’s layout): the first five commands change only rows in the database and send nothing to the bucket. Writing 4 KiB into app.db uploads one new 4 KiB block object and records in the database which range it covers.
What an object can’t hold§
Modes and owners. An object has no mode bits and no owner in the POSIX sense, so a file system has to keep them somewhere else or do without them. Mountpoint does without: modes are set when you mount, and chmod and chown aren’t supported. S3 Files keeps owner, group, and permission bits as user-defined object metadata, and reads them back when it imports objects.From Mountpoint’s semantics document and AWS’s guide to how S3 Files synchronization works. That works, with two costs. User-defined metadata is limited to 2 KB, and “after you upload the object, you can’t modify this user-defined metadata. The only way to modify this metadata is to make a copy of the object.”From Working with object metadata in the S3 user guide. So chmod on a 10 GiB file means a server-side copy of 10 GiB.
Symbolic links. A symbolic link is a small file whose contents are a path, flagged so the kernel follows it. S3 has no such flag, so each product picks a representation. S3 Files supports links and stores each as an object. Its limits page caps a link’s target at 4,080 bytes, but AWS doesn’t document how the object encodes the link.From the S3 Files page on unsupported features, limits, and quotas. Mountpoint doesn’t support links. Whichever choice a product makes, a plain S3 reader sees a short object and can’t tell it from a regular file without the product’s convention.
Hard links. A hard link is two names for one file. A bucket has only keys, and two keys are two objects. Writing both keeps them equal until the next write through one name, which the other object never sees. S3 Files and Mountpoint don’t support hard links.
Empty directories. A prefix exists only while some key carries it, so a directory with no files has no presence in the bucket. The S3 console works around this by creating “a 0-byte object” whose key is the folder name “plus a trailing forward slash.” Mountpoint takes the opposite approach: a new directory is local to that mount and isn’t preserved across a restart until a file is written into it.From the S3 guide to folders and Mountpoint’s semantics document. A program that runs mkdir -p build/out and expects it to be there after a remount depends on which choice the product made.
Rename is a copy and a delete§
S3 general purpose buckets can’t rename an object. S3 Express One Zone, a different bucket type, can rename a single file, but no bucket type can rename a directory.From Mountpoint’s semantics document. A file system with one object per file renames a file by copying the object to its new key and deleting the old one. CopyObject handles up to 5 GB in a single atomic action. A larger object needs a multipart copy, built from UploadPartCopy requests.From the CopyObject API reference.
A directory rename repeats that for every object under the old prefix. AWS’s description of S3 Files puts it directly: “when you rename or move a directory, S3 Files must repeat this process for every object that shares that prefix.” Until it finishes, “the S3 bucket temporarily contains objects under both” prefixes, and objects written to the old prefix during the rename aren’t moved.From AWS’s guide to how S3 Files synchronization works. On the mount, the rename returns at once. In the bucket, it’s a job.
A directory of five files is renamed from runs to archive on a mount that keeps one object per file. The command returns and the mount shows archive at once. In the bucket, each object is copied to its new key and then the old key is deleted, one file at a time: 10 requests for 5 files. Another reader that lists the bucket midway sees archive/1.json to archive/3.json alongside runs/4.json and runs/5.json.
The cost grows with the number of files, not the number of renames. S3 Files warns when a file system covers a prefix so large that “a rename can take up to 4 hours (approximately up to 12 million objects).” That works out to about 12,000,000 ÷ 14,400 s, or roughly 830 objects per second. At that rate, renaming a directory of 100,000 files takes about 2 minutes. In requests, it’s 100,000 copies at S3 Standard’s $0.005 per 1,000 PUT, COPY, POST, or LIST requests, which is $0.50, and 100,000 deletes, which are free.Request prices for S3 Standard in US East (N. Virginia), from the AWS Price List published October 15, 2026. AWS’s pricing page states that DELETE requests are free. The money is small. The window, during which a bucket reader sees a half-renamed tree, is what matters to anyone reading the bucket directly.
Listing is a prefix scan§
To list a directory, a file system with one object per file asks S3 for the keys that start with that directory’s path. ListObjectsV2 returns “up to 1,000” keys per request, in lexicographic order in a general purpose bucket. With a delimiter of /, keys that continue past the next slash are rolled up into one CommonPrefixes entry, so ls src/ sees src/lib/ as one subdirectory rather than every file inside it. If more results remain, the response says IsTruncated and carries a NextContinuationToken for the next request.From the ListObjectsV2 API reference.
The following figure lets you send those requests yourself, with a page size small enough to show the chaining.
The command on the mount
Page size
Every key, in sorted order
the scan starts at the first key with the prefix
Request 1 of 2
GET /?list-type=2&delimiter=/&max-keys=3
Contents
CommonPrefixes: subdirectories, one entry each
IsTruncated: trueNextContinuationToken: …1
ls / takes 2 requests, one after another: each needs the token from the one before. The delimiter folds everything under a subdirectory into one entry, so a listing costs what the directory holds directly.
Ten keys sit in a bucket in sorted order: app.db, build.sh, four keys under node_modules/, notes.md, and three keys under src/. ls / sends a LIST with a delimiter and gets back three files and two common prefixes, node_modules/ and src/. ls src/ gets back two files and one common prefix, src/lib/. find src/ sends no delimiter and gets back all three keys under src/. du node_modules/ gets back all four keys under it. With a page size of 3, any result longer than three entries takes more than one request, and each request carries the continuation token from the previous response.
A delimiter keeps ls proportional to what one directory holds directly. A recursive walk has no such help. Installing the dependencies of one Next.js app with npm ci writes 18,602 files.From Archil’s published file system benchmark. Anatomy of a file create over a network counts the round trips the same install pays. Listing every key under that node_modules/ takes at least 18,602 ÷ 1,000, rounded up, which is 19 requests, and they run one after another because each needs the token from the one before. Tools such as du, find, and git status walk trees like this. Products that put a file system in front of a bucket therefore keep their own copy of the listing, and that copy is where staleness comes from, as the next section explains.
Consistency, before and after December 2020§
For most of S3’s history, a write could be accepted before every reader could see it. On December 1, 2020, Jeff Barr described the old behavior as “a small time window where the data has been accepted and durably stored, but not yet visible to all GET or LIST requests.” That was hard on file systems, which assume that a file created a moment ago shows up in the next directory listing. Big data teams built workarounds: Amazon EMR built EMRFS Consistent View, and Hadoop developers built S3Guard.From Jeff Barr’s post Amazon S3 Update – Strong Read-After-Write Consistency on the AWS News Blog. S3Guard is worth a look, because it’s the same design this page reaches later. It kept a consistent database as “the store of metadata about objects in an S3 bucket,” to compensate for problems such as “newly created objects excluded from directory listings.”From the Apache Hadoop page on S3Guard, which records its removal from the source in 2022.
The same day, AWS announced that “Amazon S3 now delivers strong read-after-write consistency automatically for all applications,” for list operations too, “at no additional cost.”From AWS’s announcement of strong read-after-write consistency, posted December 1, 2020. Since then, any GET or LIST that starts after a successful PUT returns sees that write. The Hadoop project removed S3Guard in 2022, because there was no longer anything for it to fix.
Two lanes, before and since December 2020. In both, a writer’s PUT returns 200 OK and the data is durably stored. Before December 2020, a GET right after an overwrite could still return the old version, and a LIST right after a new object was written could leave it out. Since then, a GET returns the new version and a LIST includes the new key.
Strong consistency settles what one reader sees after one write. It doesn’t give a file system the rest of what it needs. S3 offers no locking for concurrent writers: “If two PUT requests are simultaneously made to the same key, the request with the latest timestamp wins.” Updates across keys still can’t be atomic, which is why a directory rename stays visible in pieces.From the Amazon S3 data consistency model. And the caches that products add on top bring a window back. Mountpoint, for example, can “return stale metadata for that object for up to 1 second, by default,” and longer when you turn on its metadata cache.From Mountpoint’s semantics document.
Writing part of a file§
An object can’t be changed in place. To change any byte of it, you write a new object under the same key, and readers see the old one or the new one. For large objects, S3 offers multipart upload: you start an upload, send parts in any order and in parallel, and then complete it. Parts are 5 MiB to 5 GiB each, except the last, up to 10,000 of them. On completion, “Amazon S3 creates an object by concatenating the parts in ascending order,” and until then the object doesn’t exist under that key. A part can also be copied from a byte range of an existing object with UploadPartCopy, without sending those bytes again.From AWS’s guides to multipart upload and its limits.
Consider SQLite changing one 4 KiB page in the middle of a 1 GiB database. Rewriting the object uploads 1 GiB, which is 262,144 times the change, as From a path to an inode works out. A multipart copy does better: copy the range before the page and the range after it on the server, and upload one part of at least 5 MiB around the page. That’s five requests and 5 MiB uploaded, still 1,280 times the change.
Each product that keeps one object per file handles this differently. Mountpoint uploads a new file in parts as it’s written, so writes “must always start from the beginning of the file and must be made sequentially,” and the object appears to other clients only after the file is closed or flushed. s3fs supports random writes, but “random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy.”From Mountpoint’s semantics document and the s3fs README. S3 Files absorbs writes on its own storage and exports a changed file after 60 seconds without writes, as one PUT of a new object version.
SQLite changes one 4 KiB page in the middle of app.db. On JuiceFS, the page becomes a new block object under chunks/, and the map from file to blocks changes in the metadata database. On Mountpoint for Amazon S3, the write fails because writes must be sequential, and app.db is unchanged. On Amazon S3 Files, the change waits until the file has gone 60 seconds without writes, and then all of app.db is exported as a new object version.
Blocks and a metadata database§
The other common design stops asking the object store to look like a file system. JuiceFS splits each file into 64 MiB chunks and each chunk into blocks of 4 MiB by default, and stores every block as an object. Its internals documentation gives the key format as ${fsname}/chunks/${hash}/${basename}, where the basename is the slice ID, the block’s index, and its size. An example from the documentation is jfstest/chunks/0/0/1_0_4194304. Blocks are immutable: “all file edits is achieved through new blocks,” recorded as new slices that cover the old range.From the JuiceFS architecture page and internals documentation. Names, directories, modes, links, and the map from each file to its blocks live in a metadata engine that you run, such as Redis or PostgreSQL.
Everything that had no translation earlier now has a place. A mode is a field. A link is a row. An empty directory is a row. A rename changes one row in one transaction, however many files the directory holds, because no key names a path. A write in the middle of a file uploads only the new bytes. Switch the earlier figure, The tree and the bucket, to “Blocks and a metadata database” and run the same commands to see it.
What this design gives up is the bucket. In JuiceFS’s words, after you write a file, “you cannot find the original file directly in the object storage.” Every reader goes through JuiceFS, either its client or its S3 gateway. The metadata engine becomes state as important as the data: the blocks mean nothing without the map. S3Guard used a separate database to correct listings. This design moves the whole namespace into one.
Another writer in the bucket§
A bucket behind a mounted file system is still a bucket. Any program with credentials can write to it while the file system is mounted. Three questions tell you how a product copes: when the mount notices the change, what happens if both sides change the same file, and what happens to writes during a rename. The following table shows the answers for the two products on this page that document them.
| Mountpoint for Amazon S3 | Amazon S3 Files | |
|---|---|---|
| When the mount notices | Metadata can be stale for up to 1 second by default; directory listings are never stale | S3 Event Notifications update files whose data is on the file system; others update on next access |
| Both sides change one file | No coordination between writers | The bucket wins; the file system’s version moves to a lost-and-found directory |
| Writes during a rename | Directory renames aren’t supported | Objects written to the old prefix during a rename aren’t moved |
In a chunked layout, the question mostly disappears: an object that another tool writes into the bucket isn’t a file to JuiceFS, because no row in its metadata engine points at it.
A worked example: Archil§
Archil uses both designs, and you pick one per disk when you create it. A disk synced to your own bucket keeps one object per file, at its path, in S3, GCS, R2, or another S3-compatible store. A disk on Archil-managed storage, the default, uses Archil’s internal format in a bucket that Archil operates, and you read it through the disk’s own S3 API or a mount.From our docs on data ownership and portability.
The synced disk is a hybrid of the two designs on this page. Its data is laid out like Mountpoint’s, so any S3 tool reads it. Its namespace lives in Archil, much as JuiceFS’s does in a metadata engine. That’s where mode bits, ownership, and extended attributes stay, and why a rename is atomic for any client that reads through Archil, even though the bucket shows partial results while the objects move. Links follow the choices described earlier: a symbolic link becomes a text file holding its target, and a hard link becomes a full copy. Writes reach the bucket in the background, combined so that several writes to one file become a single PutObject. They’re safe before then, because Archil replicates them across availability zones before fsync() returns.From our docs on data sources and consistency.
In the other direction, Archil keeps a cache of the bucket’s contents on its servers to make reads fast, so an object written straight into the bucket appears on the disk after a few minutes. If a program writes a file through the disk while another writes the same object in the bucket, the result is undefined and can include lost or corrupted data. The disk’s own S3 API, which reads the disk directly, supports changes from both sides at once.
A pipeline outside Archil copies batch.csv into the bucket, and the mounted disk lists it a few minutes later, once its cached view of the bucket refreshes. Then an agent on the disk and a pipeline on the bucket both write report.md. That result is undefined, and data can be lost.