Where does your data live?

An agent finishes a run and leaves data/events.parquet behind. If that file is an ordinary object in your own bucket, an analyst can query it with DuckDB that afternoon, from a laptop that has never heard of your agent platform. If it’s numbered blocks that only one client can reassemble, or a snapshot in a vendor’s account, the analyst needs a mount, an export job, or a ticket first.

The same difference comes back the day you leave a vendor. With ordinary objects in your bucket, leaving means pointing the next system at the bucket. With anything else, it’s a migration: restore, mount, copy, and check. “Your data is in S3” doesn’t tell you which case you’re in. The following figure shows what one agent workspace leaves in object storage on each kind of storage. Pick one to see what your bucket holds, what other tools can read, and how you’d leave.

Other tools reading the files§

Three things decide whether a tool you already run can read an agent’s files: the format, the account, and the key. Only one combination works without help: ordinary objects, at their paths, in an account you control. DuckDB can query s3://acme-agents/data/events.parquet where it sits, and so can Spark, Athena, a training job, or aws s3 cp.

The other combinations are common. AWS stores EBS snapshots in S3, “in S3 buckets that you can’t access directly.”From the Amazon EBS snapshots guide. JuiceFS writes to your own bucket in its own format, and its docs say that once you’ve written a file, “you cannot find the original file directly in the object storage.”From the JuiceFS architecture page. Its S3 gateway can serve the files over the S3 API, but that’s JuiceFS reading them for you. Sandbox vendors keep workspaces on their own storage, in their own accounts. Fly.io backs a Sprite’s disk up to object storage when the Sprite goes idle, and Vercel keeps a sandbox’s file system snapshot for 30 days after last use by default. Each is a reasonable product. Each also turns the sandbox into a separate data island that you copy into, and back out of.

Ordinary objects in your bucket still don’t carry everything a file system holds. An object has no mode bits and no links, and S3 has no rename, so every product that writes ordinary objects decides what to give up. Mountpoint for Amazon S3 gives up what it can’t write: it writes files sequentially, supports neither symbolic nor hard links, and renames files only in S3 Express One Zone.From Mountpoint’s semantics document. That suits training data. It doesn’t suit an agent, because git renames files into place and SQLite rewrites pages in the middle of a file. For more information about how each design maps files onto objects, see How it works.

Leaving: a copy or a migration§

Leaving is where the format turns into time and money. The following list shows what leaving involves for 500 GB of agent workspaces:

• Ordinary objects in your bucket (a bucket-synced Archil disk, Mountpoint, or S3 Files): the files are already there. Stop writing, let any sync finish, and point the next system at the bucket.
• An Archil-managed disk: copy it out over the disk’s S3 API, or mount it and copy it with rsync -aH to keep owners and links. Transfer out of Archil costs $0.07 per GB, so 500 GB × $0.07 = $35.
• JuiceFS: mount the volume with JuiceFS, with its metadata engine still running, and copy every file out through it.
• A block volume or a sandbox vendor’s disk: restore and attach the volume, or export through the vendor, and then copy.

What stays behind§

Even the best case carries less than the file system held. The following table shows what reaches your bucket from the three products in this chapter that write ordinary objects.

So an S3 tool reading a bucket-synced Archil disk sees a shell script without its executable bit, and a symbolic link as a short text file. A system you point at that bucket later inherits the contents, not the modes or the links.Sources: Mountpoint’s semantics document; AWS’s guide to S3 Files synchronization and its limits page; our data sources page.

The other formats tie you more tightly. A JuiceFS bucket is readable only with JuiceFS and the metadata engine you run beside it, so losing the engine loses the map to every file. A block snapshot is readable only after a machine in that provider’s cloud attaches it. A sandbox vendor’s storage lasts as long as the vendor’s retention rules.

Two kinds of Archil disk§

You choose the format when you create an Archil disk.

The default, an Archil-managed disk, stores data in Archil’s internal format, in a bucket that we operate. That format is what lets us checkpoint a disk in 13 ms and branch it in 33 ms at the median, whatever its size. You can’t point your own tools at that bucket. You read the disk through its own S3 API, which is strongly consistent with the file system, or through a mount, and both are served by us.For more information, see Data ownership and portability and the S3 API in our docs. For checkpoints and branches, see Can you fork it?

A bucket-synced disk points at your bucket in S3, GCS, R2, or another S3-compatible store, and writes each file there as an ordinary object at its path. A write is safe once fsync() returns, because by then it’s replicated across availability zones. The bucket receives it later, in the background, and you can’t force that sync. Deleting the disk leaves the objects in your bucket, and removing the credentials you granted cuts us off.

Two limits are worth stating plainly. The disk you start on by default is the one your tools can’t read without us. And when another program writes a file in the bucket at the same time that an agent writes it through the disk, the result is undefined and can include lost or corrupted data. S3 Files has a rule for that case: the bucket wins, and the file system’s version moves to a lost-and-found directory. We don’t, so write different paths from each side, or send those writers to the disk’s own S3 API.

A test you can run§

A vendor’s architecture diagram won’t show you what lands in the bucket. A five-minute test will. On the vendor’s storage, build a small tree that uses what agents rely on:

mkdir -p check/sub check/empty && cd check
printf 'hello\n' > a.txt && printf 'x\n' > sub/f.txt
chmod 755 a.txt
ln -s a.txt link.txt
ln a.txt hard.txt
mv sub renamed
sync

Wait as long as the vendor says a sync takes. Then, from a machine with none of the vendor’s software and only your own credentials, look at the bucket:

aws s3 ls --recursive s3://your-bucket/check/
aws s3 cp s3://your-bucket/check/a.txt -
aws s3api head-object --bucket your-bucket --key check/a.txt

Check each file against the earlier table. Is a.txt there, under its own name, and does head-object show its mode anywhere? Is link.txt missing, a text file, or something else? Is hard.txt a second copy? Did empty/ survive, and is anything left under sub/? If the listing is empty, or full of numbered blocks, your tools can’t read these files without the vendor.

Then ask for the exit in writing: if you cancel tomorrow, which files remain, in what format, and in whose account.