First published on X on October 16, 2026.
I know that we’ve been talking a lot about agents over the past few weeks, but I wanted to take some time to talk about some of the deep engineering work that Archil is doing to support our largest customers, analytics platforms. This is really important work for us to do early because we know that while agents are small data users today, they are going to rapidly and exponentially expand the amount of data they need and be the analytics customers of tomorrow.
We have always done this work by collaborating deeply with our users to make sure that their applications run perfectly on the platform, building performance optimizations and extending POSIX where needed in order to give them the performance that they need. If you have a big data application that you’d like to move to Archil after reading this, please feel free to reach out—my DMs are always open.
Today, let’s explore how we’ve done this for customers who are looking to move off of legacy systems with attached-NVMe caching capacity because they don’t handle the burstiness of agentic workloads as well as running a stateless analytics service.
State of the art today
I’ve said many times over the years that every data-intensive company is internally rebuilding something that looks like Archil in order to manage the price vs. performance tradeoff of high-speed, expensive SSD storage vs. low-speed, cheap HDD storage (S3). Today, this normally looks like the following:
You usually have some kind of a data faucet bringing data into the system, this could be application telemetry or it could be some kind of CDC stream from a transactional database like Postgres. Your ingestion engine is a microservice so that you can scale it for the amount of traffic coming into the system (separately from the query volume coming into the system). Usually, you’re getting lots of little changes from the data faucet (inefficient to plop into S3 directly), so you coalesce and batch them on a local disk before periodically pushing them to object storage.
On the query side, you also have a microservice (so you can scale with the number of incoming user queries) which farms out the queries to the underlying storage. If your user is only querying old data, then this is simple—go to S3, grab the data, run the analysis, and report the result. If your user is querying newer data, it’s a bit more challenging, you need to find the exact ingestion box where the data is being stored locally in order to find the freshest data and potentially combine that with what’s in S3.
Now, even though analytics as a field is running a “needle in a haystack” problem—meaning that each actual activity (ingestion or query) is only touching a very small amount of your overall data (which might stretch into 100 of PBs)—some of that data is often accessed much more than other pieces of data. For example, if you’re storing timeseries observability data, then a user who queries across a time range for logs (say, due to an ongoing incident or a post-hoc analysis) is often going to query that same time range again and again. This means that you users will see faster queries (and you will see reduced request costs to S3) if you do some sort of caching on local SSD.
If you do this, then every query looks the same:
- Find the appropriate data from somewhere in the system
- Bring it into the local disk
- Use it on the local file system
- LRU your disk to keep it under capacity limits
The problems
There are two critical problems that fall out of this architecture.
One is that the performance of the queries is reliant on some kind of local cache, which could have locality issues, but definitely has cold-start issues (if you launch a new server, there’s no cache, and all queries on the server perform poorly until it’s warmed).
The second is that this whole system relies on local SSD which doesn’t play well with containerized systems. During my brief tenure at Netflix, I saw not one but two multi-year projects come to a screeching halt because teams were trying to containerize these kinds of analytics problems which traditionally relied on a local disk, and there wasn’t a great choice available in the container world. This is also ends up becoming a stateful system, which isn’t very compatible with the burstiness of agentic workloads that these companies are seeing.
[aside for the nit-pickers: Yes, K8s allows you to access the local disk as a PV, but this isun’t usually desirable in a large, multi-tenant container platform for fairness and scheduling reasons. And, yes, you could use networked block storage like EBS to do this, but you run into a management nightmare (how do you always make sure you have exactly the right number for the running pods) and a cost nightmare (congratulations, your IOPS just 10x in cost).]
There is, however, an elegant solution to both of these problems (you’ll never guess it!)—shared storage. Imagine having a pool of storage that any pod could access, and was as fast as block storage. This would allow you to build a shared cache across all of your pods which doesn’t suffer from management issues, cost, or cold-start problems and it allows you to place your newly ingested data on the same system so that your query services don’t need to route to the exact ingestion machine to get fresh data.
Better yet, if you’re already pulling everything into local disk from S3 for each query, you don’t have to change any of your querying code. Let’s take a look at what this looks like.
A shared storage architecture
In this world, the shared storage service (Archil) allows all of the application services (ingestion and query) to become stateless, which makes them much easier to build on top of containers services like Kubernetes.
Ingestion no longer needs to coalesce and batch locally, it can send each request directly to disk, and allow the disk to batching to S3 as required. The query service doesn’t need to operate by finding the data on an individual host, because all of the ingestion data is already resident on the disk, and the disk itself operates as a high-speed cache over the data in S3.
This is great! This vastly simplifies how our system needs to think about storage because everything can just live in a bottomless, high-performance storage system that’s accessible from all of our services. Our query engine can drop the piece that downloads the data from S3, and just treat our service as if all the data is local all of the time.
The rub: POSIX
Okay, so this sounds great, right? Why doesn’t every do this? The problem here is simple: POSIX makes the semantics of querying really, really hard because of the limited set of primitives that it exposes. Let’s compare the different API shapes of opening a file in S3-land vs. in file system land.
Now, if you’ve been storing your data warehouse correctly in something like S3, you’re probably partitioning your objects into several different layers—sometimes a kind of object, usually a date and time, then finally some kind of per-server part file that you actually search.
S3 stores objects by mapping the full string “key” of where the object lives to the data itself. The result of this choice is that there’s no directories in S3 (you can’t atomically rename an entire folder of things in S3), BUT you can do point-lookups at any depth into the key space because it’s basically a big hash map.
POSIX and file systems don’t work this way. They are trees with real intermediate nodes for each folder along the way. As a result, they need to actually traverse the whole tree before getting to the place that they want. If a user requests to open a file that’s deep in the directory tree, Linux needs to first find each intermediate directory, validate the user has permissions in those directories, and then move down the stack.
[Interesting side note: this could be an artifact of being a FUSE file system, and as we continue to develop the long-awaited Archil kernel module, this could be something that we can completely side step inside the kernel itself by supporting point-lookups.]
Now, on a local disk, turning a single OPEN call into a serial set of reads from the disk is manageable—those reads are tens to hundreds of microseconds. On remote storage, this starts to become noticeable. But, if, like Archil, you’re actually synchronizing this data from the customer’s S3 bucket—this becomes catastrophic if each of these serial operations needs to spend 30–100ms checking for the directory’s existence inside of the object storage system.
The next problem has something similar of the same shape. When you call READ to read data off of the file system, the Linux kernel is going to chunk up that read into a lot of little tiny pieces and fulfill those reads mostly sequentially.
Once again, if you’re trying to read a large amount of data in one go—S3 is going to fulfill that request in one-shot (you pay the 30–100ms for the time to first byte, and then you transfer the data at about 80 MB/s) while Linux is going to ruin our performance by making lots of little reads (in serial) which each get hit with the same remote call penalties.
Now, this should basically highlight to you why developers continue to prefer S3 as a programming model. You call S3. It does exactly the thing you want. You build your application to request what you want. Simple.
Linux, on the other hand, doesn’t cooperate even when your application knows exactly what you want—deciding for itself how to split and chunk things up on your behalf, often causing performance problems.
What’s the solution?
Moving beyond POSIX: libarchil
Despite what you all may think of me, I’m not married to POSIX. I think that having POSIX compatibility is the price of entry into supporting the agentic world in which agents want to run “npm” and “git”, but our hypothesis is that applications which want high performance would be happy to opt-in to higher performance through an alternative set of APIs.
This is exactly what we’ve done, with the release of a piece of software that we call “libarchil”.
libarchil is a new piece of software that customer applications can opt-in to using to make calls directly into the Archil client using non-POSIX semantics, in some cases bypassing the kernel altogether. This enables users of Archil to not only bring existing applications that work against the Linux API to Archil but to also build higher performance, lower-latency applications than are possible.
In the fullness of time, we intend for libarchil to be a drop-in replacement for “stdio” which allows users to get an optimized path into the file system that avoids the nonsense associated with the kernel altogether—including new primitives for OPEN, READ, and other common operations.
Today, however, we’re focused on helping our customers, and we’ve started with two key APIs: preOpen and preRead.
preOpen(filePaths []string) error
preRead(reads []struct {
filePath string
offset int
size int
}) errorThese APIs make it possible to the Archil client, without the kernel interfering, that the application intends to call open() on a deeply nested path, allowing our client to perform all the required LOOKUPs in parallel or about to call read() on a large number of file segments, optimizing caching for those segments.
It turns out that many analytics applications find it easier to know what they’re about to read than to edit the underlying query engine itself, and we’re happy to support people with those needs. libarchil makes it possible to deliver both extremely low-latency with the throughput characteristics that people love about S3.
Okay, but what about the local SSD comparison.
It turns out that this architecture totally works. Let’s look at a CDF of latency for the queries going to an analytics engine.
Apologies, I know this is a vastly simplified model. Play around with the calculator yourself here: https://archil-latency-calculator.vercel.app/
Now, as we all know, local NVMe is faster for the p0 query (because it gets routed to the right host, is able to take advantage of the local cache, and never needs to leave the box). The problem is that, as your query fleet grows, and your data set grows, the probability of the query that needs that data being routed to the correct host is very, very low. This means that you only have a small percentage of queries that can take advantage of these low latencies before they have to hit the underlying object storage solution to get the data they need.
Once you hit object storage, you lose, because you’re always paying 30 to 100ms for the query to happen.
Archil, on the other hand, flips the script. The p0 latency is much much higher than NVMe (of course) because it needs to pay at least 1ms to cross the network. However, the majority of the queries don’t have to hit S3 because all query hosts can take advantage of the shared SSD cache, meaning that you actually increase performance for a large number of your users by moving off of local disks. Wild.
In the tail case, assuming that the application is able to effectively call preRead on what they’re about to use—and you don’t have Linux degrade you into serial network fetches—then Archil matches local NVMe. You’re going to object storage, after all, and this turns out to be the longest part of the operation.
We built Archil to break the laws of physics in the minds of our customers. In today’s world about data latencies, nobody would consider moving from local NVMe to remote storage to reduce latencies, but that turns out to be exactly the right choice for many workloads. Weirder still, using a shared SSD cache turns out to be cheaper than local NVMe because you don’t have to recache the same bytes over and over again.
This is only the beginning of what we’re able to do with Archil and, now, with libarchil. This works great for analytics applications of all kinds, but I’m excited for the (near) day that Archil can beat a metal database provider with our remote storage. That will really confuse some folks.
libarchil is available today via invite for customers running large-scale, analytics applications on >1 PB of data. Please DM me if you’re in this camp, and we’d love to give you access to try it.