How to give each Modal sandbox its own copy of a large dataset without duplicating it

To give each Modal sandbox its own writable copy of a large dataset without duplicating it, keep the dataset on a disk that supports copy-on-write branches and mount one branch per sandbox. With Archil, checkpointing the disk and creating a branch each take tens of milliseconds regardless of its size, and each branch mounts in a Modal VM sandbox as an ordinary directory that only that attempt changes.

How to give each Modal attempt its own copy of a large dataset

Parallel attempts need to start from the same state but have their own files to edit. One attempt may rewrite a file while another tries a different change to that same file. If both write to the same copy, the experiments are no longer independent, and copying a dataset of hundreds of GB once per attempt costs time and storage before any attempt starts.

The approach here is to load the dataset onto an Archil disk once, take a checkpoint, and create one branch per attempt. Each branch starts from the checkpoint and mounts as an ordinary directory. Writes on a branch don't change the source or any sibling branch, and two branches of the same disk can be mounted on separate machines at the same time, so each Modal sandbox gets its own writable copy without a full copy being made first.

What Modal snapshots and Volumes cover

Modal already has a copy-on-write tool for the sandbox itself. Its sandbox snapshot docs describe Filesystem Snapshots as copies of a sandbox's root filesystem, calculated as the difference from the base image so that only modified files are stored. Many sandboxes can start from one snapshot, each with an identical copy, to test different changes independently.

Modal also says a Filesystem Snapshot covers only the root filesystem: mounted Volumes are not included. A dataset kept on a Volume is therefore not forked by a sandbox snapshot. Filesystem and Directory Snapshots are Images that expire 30 days after creation by default, and the TTL can be changed or turned off.

Modal Volumes are shared rather than forked. Changes are committed, explicitly or by background commits, and a running container reloads the Volume to see changes made since it mounted. Modal says concurrent changes to the same file should be avoided and that the last write wins.

If the dataset fits in the sandbox's root filesystem and the attempts only ever run in Modal, Filesystem Snapshots can be enough. A branchable disk helps when the dataset lives on separate storage, when the results must be readable outside Modal without starting compute, or when the same branch should also mount on other machines.

Load the dataset once, checkpoint it, branch per attempt

Copy the dataset onto an Archil disk once, through a mount. Archil-managed storage is the default, so this doesn't require a bucket of your own. Once the data is there, create a named checkpoint to fix the starting state.

An Archil checkpoint is a named, point-in-time snapshot of a disk, and the state it references never changes. A branch is an independent, writable fork that starts from a checkpoint. See Branches and checkpoints.

Archil's limits page lists 13 ms to checkpoint a disk and 33 ms to branch from a checkpoint at p50, and says those times don't depend on disk size. They don't include loading the dataset, running the attempt, or reading files afterwards. In Archil's own measurement inside a Modal sandbox, checkpointing, branching and mounting the branch took about 1 s in total, with a 1 GiB file intact; the details are on the Archil vs Modal comparison.

If the dataset already lives in your own S3, GCS or R2 bucket, an Archil disk can sync with it. Bring your own bucket is available on paid plans, but checkpoints and branching on such a disk are an Enterprise feature. On Archil-managed storage they need no bucket.

Example: three attempts on three branches in Modal VM sandboxes

archil mount uses FUSE, which Modal supports in VM sandboxes (runtime="vm"). Modal Functions and sandboxes on the default runtime can't mount FUSE; they can read and write files on the disk with the Archil Python SDK (get_object and put_object) instead. See the Modal guide. Mounting a disk on compute outside Archil needs an Archil paid plan.

First, from any machine with the disk mounted at /mnt/dataset, create the checkpoint and one branch per attempt:

archil checkpoints create /mnt/dataset baseline
for i in 1 2 3; do
  archil branches --region aws-us-west-2 create dsk-0123456789abcdef attempt-$i --from-checkpoint baseline
done

Then start one Modal VM sandbox per branch. The image installs the archil client once, and the disk's mount token lives in a Modal Secret named archil-mount:

import modal
 
app = modal.App("dataset-attempts")
image = modal.Image.debian_slim().apt_install("curl").run_commands("curl -fsSL https://archil.com/install | sh")
 
SCRIPT = """
set -e
mkdir -p /mnt/data
archil mount "$ARCHIL_DISK_ID" /mnt/data --region "$ARCHIL_REGION" --branch "$BRANCH" > /dev/null
trap 'archil unmount /mnt/data > /dev/null' EXIT
# Your attempt runs here and may rewrite any file under /mnt/data; only this branch changes.
ls /mnt/data
"""
 
@app.local_entrypoint()
def main():
    sandboxes = [
        modal.Sandbox.create(
            "bash", "-c", SCRIPT,
            app=app,
            image=image,
            runtime="vm",
            secrets=[modal.Secret.from_name("archil-mount")],
            env={"ARCHIL_DISK_ID": "dsk-0123456789abcdef", "ARCHIL_REGION": "aws-us-west-2", "BRANCH": f"attempt-{i}"},
            timeout=60 * 60,
        )
        for i in (1, 2, 3)
    ]
    for sandbox in sandboxes:
        sandbox.wait()
        print(sandbox.returncode, sandbox.stdout.read())

The exit trap matters. The Archil client buffers recent writes, so the script unmounts before the sandbox exits, and archil unmount returns once every write has reached the disk.

Keeping the best result

Score every branch the same way, for example by mounting each one and running the same evaluation script. A shared starting point makes the attempts comparable at the beginning, and a shared evaluation gives them a common test at the end.

The winning branch is already a complete disk state. Keep using it, or checkpoint it and create the next round's branches from that checkpoint, passing --from-branch with the winning branch's name so archil branches create finds a checkpoint that lives on a branch. The original checkpoint never changes, so a new round can also start from the original baseline at any time.

For the same pattern outside Modal, see Fork a large dataset copy-on-write for parallel agent attempts.

Limits to plan around

Deleting individual branches and checkpoints isn't supported yet, so name branches per experiment rather than per run.

A branch is not a security boundary. Credentials cover the whole disk and all of its branches, so give untrusted tenants separate disks rather than separate branches.

Up to 5,000 clients can connect to one disk, per the limits page.

Put the disk in the Archil region nearest your Modal sandboxes. The Archil vs Modal comparison is candid about why: in its test, a Modal Volume read a 1 GiB file from a new sandbox at 624 MB/s, against 331 MB/s for an Archil disk about 40 ms away, where every cache miss crossed that distance. Branching removes the need to duplicate the dataset per attempt; the attempts still have to read their files.