To share a dataset of hundreds of GB across parallel sandboxes, store it once on a shared network filesystem and mount it read-only in every sandbox instead of copying it into each one. With Archil, that means one dataset disk, mounts that take tens of milliseconds, and reads served from a cache near the compute.
How to share a dataset of hundreds of GB across parallel sandboxes
The key is to separate the lifetime of the dataset from the lifetime of the sandboxes. A dataset can remain useful across many runs, while an ephemeral sandbox exists only long enough to do its job. Putting the full dataset inside each sandbox ties those two lifetimes together. When the sandbox goes away, its local copy goes with it. Shared storage breaks that link by keeping the input outside the compute that comes and goes.
Read-only mounts make this arrangement useful for parallel work. Each sandbox gets a directory it can read, but its code cannot modify the dataset through that mount. A practical layout gives each sandbox a small writable disk or directory for its outputs, leaving the large dataset on the shared disk.
This also preserves the interface most tools already expect. Scripts and agents can open files, list directories and grep what is in them without making S3 API calls. The application sees files rather than a storage integration project. That is the appeal of a shared network filesystem here, although access still has a cost. Attaching a disk is one operation, and fetching the data a job actually reads is another.
Why copying the dataset into every sandbox breaks down
Copying a dataset of hundreds of GB makes every sandbox pay to download it all before work can begin. More parallel workers thus mean more copies of the data to transfer as well as more compute. Egress and local disk costs grow with the number of sandboxes, even when they all read the same input. The duplication buys no new data.
Ephemeral sandboxes make that duplication a recurring expense. They throw away their local copy when they stop, so the next run downloads the dataset again. A successful job does not leave that copy available to the next sandbox. The compute is designed to come and go, but its preparation starts over each time.
Local copies also create a version problem when the source dataset changes. Unless all the sandboxes are rebuilt, their copies can drift apart. Jobs running in parallel can then differ both in the work they are given and in which version of the input they get. Copying is slow, and every copy becomes another version to keep aligned.
Object storage avoids distributing a full local copy, but it introduces a different interface. Most tools, scripts and agents expect a directory. A bucket can hold the data without giving those tools the file access they expect. That makes the choice more specific than “local disk or remote storage.” The useful combination is remote storage with a filesystem interface, so the input does not have to travel wholesale just to become readable.
Mount one disk read-only in every sandbox
With Archil, the recommended setup is to load the dataset onto one disk once, then mount that disk in each sandbox. A normal archil mount is exclusive, with one client able to write. Any number of other clients can mount the same disk using archil mount --read-only and read it at the same time. Those readers do not need separate dataset disks. The mount mode sets what each job can do with the shared input, so the job need not be trusted to behave politely around writable files.
Archil's published limits page gives a mount time of about 50 ms at p50. That allows a sandbox to attach the dataset at startup instead of downloading hundreds of GB before beginning work. A quick mount makes the directory available, while the job's file access determines what data must be fetched.
The storage system keeps the data in use in a durable, multi-Availability-Zone cache, where reads have sub-millisecond latency. Each client also has its own in-memory cache, so repeated reads stay local. The shared cache serves the data in use near the compute, while the client's cache holds data for that client to read again. Neither cache needs the whole dataset to be downloaded when a sandbox starts.
The client fetches data on demand as files are read. A job assigned one shard therefore does not wait for every other shard to arrive before it can start. This fits parallel work particularly well because the dataset may be large even when an individual job's input is not. The unit of work can stay small.
Example: fan out parallel sandboxes over one shared dataset
Archil's serverless execution runs each archil.exec call in its own Firecracker microVM, with the requested disks mounted under /mnt/archil/<key>. Passing readOnly: true makes the dataset mount read-only, so code in the microVM cannot modify it. A subdirectory option can also limit a mount to part of the disk. This gives the mount a defined scope. In the example below, the full dataset disk is available, and each command selects a shard by its path.
The first call lists the shard directory and turns the output into the set of jobs. Promise.all then fans those jobs out, with each archil.exec call running in its own microVM. There is no sandbox lifecycle to manage for that fan-out. Every call refers to the same dataset disk. The example counts the lines in each shard and prints the results it gets back.
import * as archil from "disk";
// The dataset lives on one disk; every sandbox mounts it read-only.
const dataset = { disk: "dsk-0123456789abcdef", readOnly: true };
const { stdout } = await archil.exec({
disks: { data: dataset },
command: "ls /mnt/archil/data/shards",
});
const shards = stdout.trim().split("\n");
// Each archil.exec runs in its own Firecracker microVM, so this fans out.
const results = await Promise.all(
shards.map((shard) =>
archil.exec({
disks: { data: dataset },
command: `wc -l /mnt/archil/data/shards/${shard}`,
}),
),
);
console.log(results.map((r) => r.stdout.trim()));
The data mount appears at /mnt/archil/data, so an ordinary command such as wc -l can consume the input without an object-storage API. Parallel execution changes where each command runs while preserving how that command opens its file. There is a runtime boundary, however. Each command can run for up to 5 minutes, so longer jobs belong in a persistent sandbox.
On another Linux sandbox or VM with the archil client installed, the mount command below sets up the same read-only mount. It makes the directory for the mount and passes the mount token environment variable through sudo. The dataset then has a filesystem path at /mnt/data. Replace <disk-token> with a disk token for the dataset disk.
export ARCHIL_MOUNT_TOKEN="<disk-token>"
sudo mkdir -p /mnt/data
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount dsk-0123456789abcdef /mnt/data --region aws-us-east-1 --read-only
Mounting the shared dataset in other sandbox providers
Archil's own sandboxes accept mounts with readOnly when they are created. Sandboxes from other providers can use the client-based approach if they support FUSE. In those environments, the setup is to install the archil client and run archil mount --read-only with a disk token. A filesystem mount needs an environment that can support it, even though the tools using the mounted directory need no S3-specific interface.
Writing changes the access requirements, but output files do not necessarily require shared writes to the dataset. The practical layout remains one read-only dataset disk shared by every sandbox, plus a small writable disk or directory per sandbox for outputs. That keeps job outputs from ever touching the shared input.
If several sandboxes do need to write to shared storage, Archil has other mount modes for them. With --shared, clients claim paths using archil checkout. With --conditional, any client can write to paths that no other client has checked out. The distinction is at the path level. These modes address shared writing, while --read-only addresses the simpler case in which many jobs consume one dataset without changing it.
When the dataset already lives in S3
An existing S3 bucket does not have to become a separate copying project before sandboxes can use a filesystem. Archil's bring your own bucket option on paid plans lets a disk synchronize with that bucket. Google Cloud Storage, Cloudflare R2 and other S3-compatible stores work the same way. The bucket remains part of the setup. This is useful when the dataset already has a home and the sandboxes need file access.
On a synchronized disk, each file remains an ordinary object at the matching key. Existing pipelines can therefore keep writing to the bucket while sandbox code accesses files through the disk. Those two interfaces serve different users of the same data. The arrangement does not require every producer to switch to filesystem writes just because the consumers expect directories.
Synchronization does introduce a timing boundary. Objects added directly to the bucket can take minutes to appear on the disk, while writes made through the disk reach the bucket asynchronously. A synchronized disk should not be understood as instant visibility in both directions. For a workflow that produces data and then consumes it, that delay is part of the handoff. The filesystem interface simplifies access, but it does not erase the time between a write on one side and its appearance on the other.
Cold reads have a separate cost as well. Reads that must go to Amazon S3 typically take 10 to 30 ms, which is slower than cached reads. The first pass over cold data therefore costs more than later passes. That is different from making every sandbox download the entire dataset before starting, because the client still fetches files on demand.