Nearly two years ago, we started Archil with a simple realization: it was clear that people were spending immense effort to build new, serverless versions of file system software on top of S3. Neon did it with Postgres, Turbopuffer did it with FAISS, and now Cursor Origin is doing it with Git. Why did it take so much effort to build each of these things?
In 2024, in our seed pitch, I repeatedly told investors that @nikitabase needed to fork and rebuild Postgres in order to build Neon, but our goal was that @archildata would make it possible to build Neon with out-of-the-box Postgres -- in an hour.
This is an immense challenge, but it's something that I'm truly delighted to get to work on every day. I wanted to spend some time to talk about how we are achieving this.
What's the shape of cloud workloads
The first thing to realize in accomplishing this vision is that actually all infrastructure workloads are exactly the same. Everything from Neon, to PlanetScale, to Clickhouse, to Vercel+webserving, to CI/CD, to sandboxes themselves are all built from the same basic stuff. [I told my friends at Clay that "Clay" would have been a better name for our company if it wasn't already taken.]
There are, of course, many parameters that differentiate the different services which leads to tradeoffs around cost, latencies, and throughput characteristics. If we accomplish our goal, though, then builders can simply specify these parameters and pop out the system that they want.
Every cloud workload is made up of three parts:
There's the compute for the workload, which is where you might run the Postgres server, the Nextjs server, etc.
There's the high-performance storage for the workload, which is usually SSD-backed (direct-attached or shared) which provides good performance at a very high-cost.
Finally, there's low-performance object storage for the workload, which provides good cost characteristics for systems which have many tenants (and most tenants are inactive).
Each of these pieces can, of course, be scaled:
-
OLTP databases like Postgres usually have a single piece of compute pointing at the storage layer, and time-lagged read-replicas.
-
OLAP workloads like Clickhouse can have massive amounts of compute pointing at the same storage backend so that they can do map-reduce queries over the data and large amounts of ingestion.
-
Sandbox workloads have a single piece of compute attached to an SSD layer, usually with transparent materialization of the data so that you can startup the server quickly.
-
Git-like workloads, like Cursor Origin, use a combination of these technologies for different pieces of the stack: S3 for the write-ahead log and SSD for the git storage.
Understanding parameters
Notably also, one of the parameters for the model is whether the service tiers to S3, and how much of that data needs to be resident on the high-performance storage before the compute service can start work.
For example, a database like PlanetScale does not do tiering to S3, and requires that 100% of the data be resident on the local SSD before it can start serving (this is the replica catch-up time that happens when a server fails or you grow your cluster). Databases like Neon do tier to S3, and don't require the data to be resident to move forward. Traditional compute services need the entire Docker image to be resident on the SSD to start, causing poor cold starts. Many sandbox companies are able to lazily materialize the data onto the compute so that the service can start while the data is still coming in.
These are all just different shapes of the same workload, with different pieces of software. If you built a storage solution which allows customers to tell you -- "I want X% of the data to tier down to slow object storage" or "I don't want to serve any traffic until the data is fully local", then you solve for all of these use cases.
If you look on the write side, there's also a parameter that tradeoff durability against the speed at which those writes happen.
For example, most sandboxes (only confirmed in E2B code), don't actually do anything when you perform a write because they assume that the data you're writing is ephemeral (don't run a database there! lol). The next step up is local disk durability, which [from what they're saying] appears to be what things like Vercel Drive do. The next rung up is actually replicated storage (where you can actually validate that your data is safe) which is either zonal (I believe with celld's new mode it doesn't care about cross-zone) or regional. Finally, the highest level of durability (and the slowest writes) are if you actually wait for S3 itself to acknowledge the write.
I believe that the only reason that people opt for the "wait for S3" durability mode is that they do not want the hassle of running a stateful SSD storage layer, it doesn't seem to be that there's any real benefit to doing this. That said, you shouldn't just "implement raft" and call it day on data safety because storing data yourself is very hard to do safely. Use a provider like Archil.
Thinking about the interface
The first choice you need to make when you're building a write-ahead log is what it is that you're write-ahead logging. The log allows you to virtualize a view of data by combining some background view (what other people call "compacted" or "read-optimized" view) with a set of commands on top of that view. To make this generic, the "view" that we want to virtualize is the bucket or the file system itself. This means that the underlying data should actually be stored like a regular file system, like on disk. It should have inodes, directory entries, and more. This is the most generic way to ensure that we can support all workloads, because we know that all workloads will fit this shape.
The second choice you need to make is what set of commands your write-ahead log actually accepts. Many people choose to do a very simple set of commands here: streams only have Append, Cursor Origin only supports "pushes". What we've done (which really irks people) is chose to support the POSIX file system API for our service out of the box. The reason for this is simple: the vast majority of software ever written (and that ever will be written) is written for the file system (including SQLite, Postgres, vector storage, even RocksDB).
Choosing to not support POSIX is a choice that ends up locking you out of this set of software forever, because POSIX is [nearly always] net-negative ROI -- and requires that people purpose-build their software against your stack. We think this is too large of an adoption blocker for a storage system, and we want to meet customers where they are.
The nice part about supporting POSIX is that it's actually a superset of S3 functionality (for example "PutObject" is just file writes and an atomic rename [+ some directory nonsense]). This makes it relatively simple to build additional APIs on top as developers need them. We've already built an S3-compatible API, and we expect to deliver even more APIs for people who are purpose-building high-performance applications (such as AppendObject and other prefetch primitives, more on this later).
What about scale
Write Scalability: The thing about write ahead logs is that they necessarily imply a total ordering of everything that happens on the system. Ordering is usually the enemy of scaleability because it implies that there's a single mutex somewhere that is responsible for appending things into the order.
There are several things that you can do to improve the scaleability of a log-based storage system, for example:
-
You can split up the work where raw data is uploaded (to parts objects in S3) from when it's appended to the log (a single CAS operation), which reduces the "critical section" that the log is locked for
-
You can group commits such that you actually commit more than one operation at a time during the "critical section"
-
You can split the log up into multiple logs working in parallel which do not provide a total ordering amongst themselves (though, of course, some operations may need to coordinate across these sub-logs).
It's clear that a generic storage system that solves for all cloud workloads needs to solve for all of these properties, by doing all of this simultaneously.
Read Scalability: Scaling your reads up is actually a much simpler proposition on a storage system, because you just need to store more copies of the data in more caches so that the data can be served from multiple locations. This reduces your problem of "how can I serve more reads" into "how can I invalidate the caches that I have scattered about the system", which is strictly more tractable.
Putting it all together
There are two important things that I want you to take away from this post.
First, I want you to recognize that all storage workloads collapse to a generic set of primitives which can be built into a single system, which can become the default way that storage works in the cloud.
Second is that this system is actually much more than just a "library" that you can add to your application.
There are millions of people who have written write-ahead logs on S3, and libraries that you can use to format that log and manage compaction, etc etc. The real value is building the entire system for the user: including a durable space to acknowledge writes before S3 (so that you can do millisecond and microsecond commit latencies), the ability to manage the high-performance caching built in, and the place to run the compute.
This hasn't really been done before, and it's where we think that @archildata will thrive.
The whole solution looks something like this:
We give our customers a custom-client which exposes the ability to run FUSE operations (soon to be kernel-mode), but also supports APIs like the S3 API, log-based APIs, and higher-performance specialized APIs like pre-fetch.
The user is able to tell us how they want to form a write quorum. In some cases, they don't care at all, and we acknowledge without doing any writes. By default, we do multi-AZ replication across our servers, but in other cases we have the ability to do PlanetScale-style "local disk + cross-AZ disk".
You tell the client how you want to support reads. You can synchronously pull your data to the local disk with high concurrency, enabling you to only serve reads from the local disk. Or, by default, materialize the data that's in the server locally -- useful if you want to avoid cold-start times.
The client has the ability to fan-out both reads and writes to the Archil servers, enabling nearly unlimited throughput for things like model training or data analytics.
Finally, the user can tell us whether or not they want their data to exclusively live in the NVMe devices on our servers. Or, by default, if they want that data to tier down into S3.
This architecture is able to scale and handle all cloud workloads built today, and it shows. We are helping customers who are doing: CI/CD, building agent platforms, run model training, OLTP databases, and do data analytics. All from a single storage system that allows them to specify the right properties for the right job!
Archil is the last WAL that you need to build on top of S3.