It feels like there are hundreds of new infrastructure companies popping up every day, and because these companies all recognize that the next set of workloads (namely, AI agents) are stateful workloads most of these companies are marketing explicitly about being able to "persist disk state" so that you don't have to worry about whether or not your data will be around if the compute stops while you're not using it.
However, not all "persistent disks" are created equal. Let's talk about how data is stored safely, what kinds of things your infrastructure provider could be doing, and what questions you should be asking to safely run persistent workloads in the cloud.
Isn't all data safe?
Like most things related to data storage, S3 and object storage have ruined the conversation by making it so easy to use storage that people have forgotten how things outside of that ecosystem work. One of the ways that object storage won the data storage world is around durability or "when data is safe after storing it".
Object storage had the innovative idea that once a call to PutObject returns successfully, it would be safely stored in the backend storage system -- for many, with "11 9s" of durability.
[side note: It's not immediately obvious to people that the "11 9s" number is per-object. This means that as you approach 100 billion objects stored, you might want to start thinking about how you're protecting that data as opposed to just letting Amazon handle it for you.]
This isn't exciting, right? Like, on my laptop if I call "cp", "mv", "git clone" -- the data is safely stored on my hard drive once those calls complete, right?
Like with many unintuitive things in file systems, it turns out that this is not the case.
You see, object storage is designed for handling raw throughput, so they want you to optimize for sending and receiving large objects, as a result, it's not a performance problem to always make the data durable when you put the entirety of these large objects. This is not, however, what file systems are optimized for.
File systems are optimized for lots and lots of small writes which could be: file creation, metadata updates, file data changes. If we were to persist each of these to the disk when they happen, it would cause a tremendous performance bottleneck as we both need to (a) wait for the disk to do this work and (b) bottleneck on the disks ability to execute raw operations per second.
Instead, the file system has a different mechanism -- fsync -- that allows the application to tell the operating system that actually it's now time to put the stuff that I just wrote onto the disk. That looks like this.
Now, for many, many reasons (including the operating system trying to push stuff to the disk asynchronously), this turns out to mostly be okay and data loss is super rare -- even on personal machines that tend to turn on and off randomly. Notably, this is [of course], why you need to "Eject" an SD card on your Mac, or a USB drive on your Windows computer. That button calls fsync() so that any in-flight writes are stored safely. If you don't do this, then you might lose the last few seconds of recent writes, but... do we care?
Why should I care if I have data loss?
There's lots of writing on the atomicity of file systems, so let's focus on a simpler problem -- databases. Every major open-source production database system (SQLite, Postgres, and MySQL) work by storing their data on top of a file system. You can actually think of databases as an API that allows you to more easily and safely work with the file system.
Notably, they are the one class of application which rigorously and correctly works to call fsync(). This is because they want to ensure a simple property: if you mutate a table (say, insert a row), when the call returns success, it should be impossible (absent drive failure) to lose the data associated with that write. This looks something like this:
Even though the operating system is only editing the database files in-memory, the database won't respond to you until the fsync is actually complete. Why is this important? Well, let's imagine that we build a faster database that doesn't call fsync to make the data safe during each mutation. Let's, hypothetically, call that database MongoDB.
This is going to be way faster on benchmarks (insertions are just in-memory operations), but on any system crash, we might lose the most recent few seconds of data. Now, in some cases, this could be benign (it's fine if we lose analytics data), but in most cases it's NOT benign specifically BECAUSE we already told the application that the write was successful.
Imagine that you're creating a new user account by inserting it into your database, your database says that it's saved, and then when the user goes to log in -- it doesn't work! Worse, what if it's an airline ticket that the Airline told you was successfully purchased, but then it turns out that they lost the ticket! Worse still, what if it's a record of medicine that's been administered to a patient?
There is a deep evil in data loss because once the storage layer (the database, in this) tells the application that the data is safely stored, any number of application-level and meatspace-level actions could occur expecting the data to be there on the next read.
How does this affect 2026?
The challenge that we see today is that suddenly so many different "neo-clouds", "compute platforms", "sandbox providers", and more are popping up to try to serve this new class of highly-stateful workloads. Each one of these companies is (correctly) making different trade-offs to serve a slightly different set of workloads better. This could be differences in networking, the way that you access the machine, the places where the machines run.
Notably, most of these providers claim that they are all the first people to build stateful compute services, by persisting the disks that you use on their services, but they aren't all forthright about how they persist this data. Depending on how they do it, it could be completely safe, or it could be a really, really bad time for your users. Worse, you usually won't know about hitting these problems until something goes wrong and you've lost data, when there's nothing to do about it.
There are basically two models here, and it completely depends on what the provider does the application calls fsync. First, the "safe model":
The safe model is how all major storage services are architected today: Archil, S3/GCS (on PutObject), EBS/Hyperdisk, and EFS/S3 Files. When the application requests that data be stored safely (by calling fsync), the storage service (either running on the machine like EBS/Hyperdisk or off machine like EFS/Archil) is going to redundantly store that data across multiple hosts, so that no individual host failure can result in data loss. This is also how services that operate on "raw NVME" like PlanetScale Metal work, with the "storage service" in this case being Vitess.
Now, the specific number of hosts that they store on, and the amount of data on each host may vary so that each service can tune its costs, throughput amplification, and durability "5 9s vs. 11 9s", but the basic model is the same. It's not possible to keep data stored safely on a single SSD safely.
Of course, this incurs a performance penalty and a cost penalty, so if you're a provider trying to win on benchmarks (like MongoDB was), you might look to implement a system that looks more like the following:
You could hypothetically just acknowledge fsync calls as soon as the changes hit the local disk, and have an on-host service kind of asynchronously push the data to a more stable storage system like S3, or a clustered storage system like Ceph.
This causes your benchmarks to speed up for sure, especially when you're running database workloads where write-speed is defined by fsync throughput. But it's not safe. Why not?
For those few seconds before it's pushed to the real storage service, data is only stored on the single host NVMe. This means that your database could have acknowledged a write to a user, the user could have done something, and a server death could erase that database write.
Now, in this architecture, there are different levels of "safety". If they actually require that the write hit the disk before acknowledgement, then a server restart might not be enough to lose data (assuming that they recover and flush the local disk when the server comes back online), you would have to have the server die.
But, folks who've worked on cloud services for long enough know that servers die all of the time. And, if you assume that users are continuously writing data to their disks, then there's always something for the server to lose when it inevitably dies.
[side note: Often, server death rates are actually higher in hyperscalers than other data centers because AWS and GCP are unwilling to replace just part of the server (e.g. if the network card fails) and you the intact disk back. Just one of the pleasures of running a storage service.]
Where does this leave us?
If you're relying on a providers persistent disks in order to keep your workloads alive, ask them what they do when the application calls fsync(). If they don't replicate it to multiple machines, like Archil does, then it's probably not a safe place to run workloads with critical data, and definitely not a safe place to run databases.