Yesterday, @ankrgyl, the CEO of Braintrust, remarked that it felt like sandboxes were the convergence of servers and serverless. He should be correct here, it is the case that sandboxes kind of feel like "serverless servers". The problem is that nobody has built an interesting, serverless version of the sandbox. Why is that, and what would it look like?
What does it mean to be serverless?
Today, it seems like every infrastructure that you buy is serverless -- databases, storage, website hosting, inference-- and lots of people have lots of opinions about whether or not "serverless" is a good thing, or even what it means. Let's define it by starting with my favorite thing -- "what was the world like before AWS".
Before cloud computing was the only way to deploy applications, we had actual data centers and colocations. In this world (with varying levels of white-glove service from your DC), your company would rack+stack computers that you select and get network access to them. You would provision the machine, install the OS, and set up your application to run. If you needed storage, then Oracle would be happy to sell you a Box that could run a database for your computers to connect to. If you needed more capacity because your business was growing, you would need to obtain budget from your department, open a ticket to IT, buy the server, drive over to install it, and then get it all set up. This could take a long time, and wasn't something that could be done in time if your product went viral on Slashdot. If one of the hard drives or the memory in your server went bad, guess who was on the hook for buying a replacement and repairing the server?
This was pretty bad, and companies like Amazon started to realize that there was a lot of "undifferentiated heavy lifting" in this process. If every company was rack+stacking servers (requiring time and expertise from each company), wouldn't it be more efficient for one mega company to do all of the rack+stacking and have everyone else rent from them? Thus, we got cloud computing.
With EC2, you could now launch additional capacity in seconds instead of in months. You were still responsible for making sure your operating system was up to date, but AWS would handle replacing hard drives and procuring actual servers. Because of the large number of servers that AWS had, they could even provide a huge amount of diversity at the click of a button (need lots of ram? needs lots of CPU? they have you covered).
EC2 instances launched so quickly and there was so much capacity that an entirely new idea was born: auto-scaling. Now, if your product went viral on Hacker News, AWS could actually monitor the servers for you and automatically launch more (in just minutes!) to ensure that your application stayed alive during a period of high-traffic.
Now, as @dhh knows the best, the unit-cost of these virtual servers is actually higher than installing the servers yourself in a colo. If you bought one server and placed it in a colo and used it 100% of the time, it would be cheaper on a per-hour basis than doing the same thing on EC2. We paid that higher unit price because most applications have pretty spikey traffic. You could save a lot of money by spinning servers and up down rather than purchasing servers for the maximum amount of traffic you ever expected, and we became familiar with graphs like these:
This was good for a time, but people started to realize that there was still that "undifferentiated heavy lifting" creeping in. It turned out that while we needed EC2/VPS for some kinds of "legacy" applications (that Oracle database still needed a Box to run on), most applications being written during this time were actually just... websites. The web was exploding! Why was it that everyone who wanted to deploy a website needed to spin up a server, install an OS (and do security patches!), configure load-balancers, and more.
Even with auto-scaling kicking happening in O(minutes), that's still minutes of virality on Twitter where your application is just down. That's not acceptable, so the industry moved into the next form: "serverless" compute.
The thinking from AWS was, "what if we just did all of this on behalf of our customers?" If that's the case, then the developer only needs to worry about the "business logic" of their application -- what their business actually uses to drive revenue.
The nifty thing here is that the infrastructure is already running on AWS's side, so rather than waiting O(seconds) to launch a server, they could start a function in O(hundreds of milliseconds). This was fast enough to unlock a new idea beyond auto-scaling: what if we only ran your code while customers were requesting it? If nobody was accessing your website, you would pay $0, and if you got requests, AWS would automatically provision -- on-demand -- enough compute to serve those requests.
Now, Lambda has been moderately successful, but I think it sort of missed the prime because it was too generic. Developers were struggling to invent ways to get around all of the weird quirks of Lambda -- a 15-minute max runtime, originally a very small amount of onboard storage, etc. The company which really began to be known as the kings of the serverless era specialized on what they ran: Vercel.
Rather than spinning up general-purpose compute, Vercel started by hosting exactly one kind of thing -- Javascript frontends. As a result, they didn't need to wait O(hundreds of milliseconds) to "start" capacity for users, it was just a function call away. Users didn't need long backend runtimes because these were literally just web apps, and users didn't need onboard storage because a whole new crop of "serverless" stateful services popped up to make building in this way easy.
This meant that "serverless" compute, like Lambda functions or Vercel deployments, appeared as nearly perfect "pure functions" which would take an input event + state of the world and produce a deterministic output across those inputs. Once it was set up, it could be used over and over again.
And, just like with EC2, there was again a per-unit price-hike. People correctly pointed out at the time that Vercel was just a "wrapper on AWS which is higher-priced". This is true, but again, customers were paying for more than they got with AWS directly. They now didn't have to worry about their application ever having downtime because capacity was infinite and always available.
Therefore, when I think of what "serverless" means, it comes down to the following things for me:
- User does not need to manage the operating system or security updates on the node
- The service doesn't "auto-scale" it automatically serves traffic for end-users on-demand by launching capacity or calling into code directly
- The user only needs to worry about their business logic -- the serverless system is nearly a deterministic function that can be applied over and over again to the same inputs to get the same outputs
Now, this leads to some sort of odd conclusions from the current state of the world. For example, which database do you consider to be "serverless"?
I don't think it's Neon.
Where does this leave us with sandboxes?
Now, we know that the end-state of the industry is that everyone will have their own sandbox company, so let's first start by looking at usual sandboxes, and then look at Cloudflare's (since they tend to march to the beat of their own drum). Usually, sandbox usage looks something like this:
You have some kind of agent that wants to run code that could destroy the system, so rather than run it on the current system, you spin up a sandbox to run that code. The user does this, it's great, but eventually they become inactive on your platform, so you pause the sandbox to save on costs. When they come back, you call resume, and things are great until you eventually kill the sandbox. Awesome! Is this serverless?
If your definition of serverless is "I can easily interact with it from Javascript," then maybe. But I don't think so. There are a couple of really interesting issues with the standard API.
First, despite @HeyGarrison 's best efforts, spinning up a sandbox still takes some time, which means that developers don't like to expose that latency directly to their end users. So, they want to spin these things up as early in the process as possible. BUT, charges start accruing as soon as the sandbox starts, which... means that it's wasteful to spin up a sandbox before the user needs it. What if they never issue a command that needs to run code (maybe I could just use just-bash)? What if they walk away for a few minutes before sending their next message?
I know lots of developers that are basically trying to home-build functions which guess whether or not a command needs a real sandbox so they can route it to either just-bash or spin up a sandbox. This is what "undifferentiated heavy lifting" looks like.
The sandboxes of today are servers, plain and simple. Let's look at our checklist.
- Does the platform take care of managing the operating system? Yes, so that's a check.
- Are they automatically spun up and down in response to demand? No, users need to manage their sandboxes carefully as actual resources to avoid charges and latency penalties.
- Are they treated like a function that you set up once and can use over and over again? No, they are bespoke environments. In fact, I've heard from developers who struggle because sometimes the AI model kills the sandbox and it needs manual remediation.
The ability to spin them up and down within milliseconds just makes them a "better EC2", but if the end-game of infrastructure is to move to purpose-built serverless solutions, these sandboxes are not them.
[aside: This also means that this API is sort of one AWS feature launch away from being undifferentiated from the Hyperscalers -- they just need to get EC2 instance launch times fast. Now, I worked at AWS for a long time, so I know how things can cross from "easy" to "impossible" on a dime, yet...]
What about suspend/resume? Isn't that serverless?
Nope. In fact, EC2 has had the ability to pause and resume (StopInstance and StartInstance) for a decade, and nobody seems to believe that "EC2 is serverless".
What about Cloudflare or Vercel? Does pricing on Active CPU help get you closer?
In my thread with Ankur, Jordan ( @acoyfellow ) from Cloudflare (correctly) pointed out that Cloudflare Sandboxes already supported much of what I was looking for in a "serverless sandbox".
This is true, to an extent. Let's look at the API and docs.
I like a lot about what I see here. I think that their docs do a good job of making it clear that you should have a shit ton of these things because creating them doesn't mean that you need to pay for them (see the call to "getSandbox" which is, obviously, per-user).
They will spin up your sandbox on-demand (check) in a Cloudflare Container, and then spin it down after a few minutes of inactivity (this is a partial check, I think that the "fluid compute" idea of directing repeated requests to a running container is fine, but I think the platform should pay for the timeout time instead of the user for it to really count).
There's really just one thing that kind of bothers me about their docs, and it will surprise you: the call to mkdir().
The sandbox is a tool, not a place.
There is something about that "mkdir" call that betrays how Cloudflare thinks about their sandboxes compared to how we thought about Lambda or Vercel deployments.
Did you actually create one Lambda function per-customer in the serverless era? Did you create one Vercel deployment per-customer? Did you upload customer-specific files to either one of these?
Of course not, because that's kind of ridiculous and would quickly become a management nightmare. The Lambda function and the Vercel deployment were, for better or worse, "stateless" over an outside set of context that the application needed access to (in most cases, an S3 bucket or a database).
Even though you know that I'm a huge proponent of the fact that state is the thing that's different about AI agents, I don't think it belongs in the sandbox layer itself, or you end back up in a world in which you're managing servers, one-by-one.
Instead, the sandbox should still be a stateless tool over some kind of external state, but it's a different kind of external state than before. The external state of today is, of course, a file system, and the sandbox should be a tool used to manipulate that file system. Sometimes this is read-only, sometimes this is writes to the file system.
In either case, I don't think that the agent harness of tomorrow will actually want a computer to do its work on. The serverless end-state of sandboxes is that they become a query language, much like SQL, that allows the agent to access the totality of software ever created in a serverless way.
In this world, there is no resource that's ever created. You're not charge for what's "running" because nothing is running. You're only charged for executing these "queries" against the context of the agent. You don't need to worry about uploading and managing files in a "sandbox" because there is no "sandbox", there is only the customer's context. And, the customer's context should support running an unlimited amount of compute on top of it as the workloads that agents run become larger and more demanding.
This "sandbox" query language will run on top of the file system, and provide a way for the harness to create and manage long-running, network addressable services (like databases) so that the agent can literally build a distributed, serverless computer in a piece-by-piece way that allows it to most easily manipulate its context.
This looks a lot more like how serverless is supposed to look, based on past patterns.
Why hasn't anyone built this yet?
Like most things, it's hard. Selling provisioned services (like you request 2 vCPU and I charge you slightly more than it costs me) is a way simpler proposition, and it doesn't even require category creation.
It's not super clear how you even go about breaking down the computer in an abstract way into a serverless state machine that continues to expose the full expressiveness of what's possible today. I do know, however, that if you do this, you will save agent builders a tremendous amount of headache because they won't need to worry about: when to launch sandboxes, if they're paying for idle resources, or even which sandbox the user data is saved in.
The sandbox will just be the tool that the model uses to manipulate context. As you may have guessed, we're working towards building this truly serverless computer at @archildata. We've built the world's most powerful file system, and we've exposed way to use Linux commands to provide models the ability to query the context on that file system. It's already saving builders a tremendous amount of time from worrying about sandbox resources, but there's still a tremendous amount to do.
If you're interested in learning about what's coming next, we're hiring. Hope to hear from you soon.