If you used an agent in the past, you've probably felt like you had to hold its hand while completing the task. It could be having to give it the right URL for a web search, specifying a file instead of a directory when asking to make a change to a codebase, or anything else to narrow down the information the agent needs to work through. When given the right context it needs to solve a problem, AI agents tend to be quite good at what they do. The problem is, getting them just the right set of information is the hardest part.
With terabytes of corpus to search through and a context window of at most about a million tokens (a few megabytes of text), finding the right data to perform the inference on is an uphill battle, a problem known broadly as context retrieval. The wrong retrieval mechanism can leave your agent editing the wrong file, wasting compute on the wrong solution, or going through the wrong SOP. In modern agent workflows, data retrieval is one of the hardest problems to overcome, sitting at the heart of retrieval-augmented generation (RAG) architectures.
How to Store?
When it comes to data retrieval in general, the underlying data storage is one of the most crucial parts. Databases excel when you have structured data and need extremely low latency, object storage is optimized for unstructured data and high throughput, and file systems sit somewhere in between, handling unstructured data while also supporting random reads and writes.
Object Storage
Object storage, like AWS S3 or GCP's Google Cloud Storage, offers virtually unlimited data storage without scaling concerns, virtually unlimited throughput, and redundancy and availability that is second to none. This makes it great for long term storage or for larger files like photos or videos. If random reads and writes are not needed, many of the drawbacks of object storage disappear.
However, with a non-deterministic workflow, random reads and writes are essentially guaranteed. Agents retrieve small portions of files and make small mutations to them, which requires querying the whole object and re-uploading when using object storage as a backend.
Object storage also gives agents few coordination primitives. Amazon S3 now offers strong read-after-write consistency for individual objects, but there is no atomic rename of a directory, no locking, and no way to update several objects in one step, so tightly coupled read-modify-write loops have to build that coordination themselves.
Agents don’t just need “more context”, they need the right context, organized in a way that matches how the work is actually structured. Agents need a logical grouping of data: "everything for this project," "the docs and configs for this service," or "the policies for this customer" — not a flat ocean of unrelated objects. Object storage has no true hierarchy, so agents require additions like a vector database or metadata index to create logical groupings.
Object storage does have its place in the agent workflow — such as storing large backups or checkpoints — but otherwise it is too slow and bulky for most agent workflows.
Database
Databases, particularly relational ones like PostgreSQL or MySQL, are the gold standard for structured data. If your data fits neatly into tables with defined schemas, you get low-latency queries, strict consistency, atomic transactions, and rollback guarantees, the properties collectively known as ACID compliance. When a bank moves money between accounts, this is exactly what you want.
The problem is that agent workloads rarely look like this. When a coding agent works through a repository, the "data" is source code: text files with irregular structure that changes by language and framework. When a research agent pulls papers, the data is natural language. None of this maps cleanly to rows and columns, and forcing it to creates friction immediately. Relational databases require schemas defined upfront, but an agent often doesn't know what it needs to store until mid-task. Its data model evolves as the work does, which means either constant schema migrations or generic catch-all columns that push all the structure and validation back up to the application layer.
Access patterns are the other mismatch. Databases are optimized for small, well-scoped operations on individual rows. Agents work with large, contiguous chunks: whole files, full documents, entire conversation histories, and multi-step reasoning traces. They ask for "the last N steps" or need a complete codebase loaded into context. These are ordered, history-oriented, append-heavy workloads, and OLTP systems treat data as unordered sets of rows, making that shape awkward to express, especially once large blobs get involved.
The common workaround is stuffing unstructured data into blob or JSON columns and using the database as a metadata index. Plenty of systems do this, and it works, but at that point you've built a worse file system on top of a transactional engine while still paying the operational costs of one. Databases absolutely have a place in agent infrastructure, tracking metadata, managing task queues, and storing anything genuinely relational. But as the primary storage layer for the data agents actually read and mutate, they force you to fight the abstraction the entire way.
Why File Systems Make Sense
File systems sit in the sweet spot between the two. They handle unstructured data natively while still supporting the random reads and writes that agents are guaranteed to make. When a coding agent needs to edit one function in a 2,000-line file, it can seek to that position and modify it in place — no re-uploading the entire object, no forcing text into a schema it was never meant to fit.
The hierarchy allows a logical grouping of all your data. Codebases, document collections, and project data are already organized as files and directories. The relationships agents care about are encoded directly in paths and structure. Agents can operate on data in its native form instead of translating it into something else first, and there's no schema to define upfront: a file can be code, JSON, or binary, and it can change shape over time without a migration.
Finally, file systems match the actual shape of agent work. Agents operate in tight loops: read context, write intermediate state, read it back, iterate. They append logs, tweak configs, and edit small files repeatedly, creating a working set with strong locality. File systems handle this naturally through low-latency access, caching, and in-place updates. It's an abstraction that's minimal and composable; you can layer indexing, versioning, or durability underneath it, and the agent's interface never changes. It just reads and writes files.
The Missing Piece
There's a catch, though. The file systems most of us reach for weren't built for this either. Local disks don't scale past one machine. Traditional network file systems like NFS or Lustre were built for long-lived servers and clusters with a fixed set of clients, not thousands of short-lived agents hammering shared state with small, frequent mutations. And your data probably doesn't live on a file system anyway. It lives in S3.
This is the gap Archil is built for. Archil sits in front of object storage (Amazon S3, Google Cloud Storage, Cloudflare R2, Azure Blob and other S3-compatible stores) and exposes it as a fast, consistent POSIX file system, so agents get the low-latency random reads and writes and the hierarchical structure they need, while the data itself stays in S3 with all the durability and scale that made you put it there in the first place. You don't have to migrate terabytes of data or maintain a synchronization layer. The agent sees files; the object store stays the source of truth.
Agents are only going to get more capable, more numerous, and require more data. The infrastructure decisions made now, while workloads are still small, are the ones that will calcify. Getting the storage layer right means giving agents an interface they were effectively trained on, backed by storage that can keep up.