To fork a large dataset copy-on-write for parallel agent attempts, checkpoint the disk that holds it and create one branch per attempt. Each attempt gets its own writable starting state without making a full copy of the dataset first. Archil creates a checkpoint in about 15 ms and a branch in about 35 ms, regardless of dataset size. Each attempt can then change its branch without changing the source or another attempt’s files.
How to fork a large dataset copy-on-write for parallel agent attempts
The workflow separates the shared starting state from the work done by each attempt. First, mount the dataset disk and take a named checkpoint. Then create a branch from that checkpoint for each attempt, and mount each branch on the machine or sandbox where that attempt will run. The checkpoint fixes the starting point. The branches provide the writable copies.
This distinction is useful when the agents must change data. An attempt that cleans a dataset needs somewhere to put its edits, while an attempt that migrates it may need to rewrite files. With separate branches, those changes belong to the attempt that made them. Only the data an attempt changes is new, which is the copy-on-write part of the arrangement.
The unit being forked is the disk that holds the dataset. That gives each attempt a writable disk state where it can make the changes it proposes. After the attempts finish, the same validation process can score their results, and the selected branch can become the starting point for further work. There is no need to treat the winner as a disposable scratch copy. It can remain the working disk state.
Why full copies of a large dataset slow parallel attempts down
Several agents cannot safely treat one writable dataset as their private workspace. If their work includes cleaning, migration or feature regeneration, their writes can overwrite one another. A file one attempt changes then differs from the starting file another attempt expected, defeating the purpose of independent attempts.
Making a full copy for every attempt solves that separation problem, but introduces a preparation cost. Both the copying time and the storage required grow with the size of the dataset, before any attempt starts. The larger the starting state, the more data must be reproduced just to give each agent a workspace. Each new attempt also needs a full copy of the same starting data.
Copy-on-write changes what must be new for each attempt. All branches start from the same state, while only the data changed by an attempt is new. The distinction is between copying the starting dataset and recording divergence from it. This does not make the agents’ actual work free. A task that rewrites data still has that work to do, but creating its branch does not require a full dataset copy first.
The performance claim therefore applies only to the fork operations. Creating a branch quickly says nothing about how long cleaning, migration, feature regeneration or validation will take. It removes the dataset-sized copying step from workspace preparation. That is a narrower claim, and a useful one. The agents can spend their time attempting the task.
Checkpoint once, branch once per attempt
Archil checkpoints are named, point-in-time snapshots of a disk. The state a checkpoint references never changes, so it provides a fixed baseline for the attempts. A branch is an independent, writable fork of a disk that starts from a checkpoint. These are different jobs. The checkpoint preserves the starting state, while the branch is where an agent can make changes.
Archil takes about 15 ms to create a checkpoint and about 35 ms to create a branch, regardless of disk size. Those timings describe the two operations used to prepare the forks. They do not describe mounting a branch or completing an agent attempt.
Writes on a branch do not affect the source or any sibling branch. Each branch can also be mounted on a different machine or sandbox at the same time, which makes this arrangement a fit for fan-out. The attempts do not have to take turns with one writable workspace. Their shared relationship is to the checkpoint they started from. Once they write, each branch records its own version of the dataset.
The branch structure can extend beyond a single round. Like Git branches and commits, it forms a tree, so a branch can be checkpointed and new branches created from that checkpoint with --from-branch. Later attempts can then start from an earlier attempt’s result, with that result taking the place of the original baseline. The fork becomes a new point of departure.
Branching is available on disks that use Archil-managed storage, which is the default for new disks. That storage requirement applies to the workflow described here. It is distinct from the choice of machine or sandbox that mounts a branch.
Example: three agent attempts on forks of one dataset
Start by mounting the dataset disk and taking the checkpoint from that mounted disk. The command is archil checkpoints create, and the checkpoint name must be unique within the branch where it is taken. In this example, the checkpoint is named baseline. It is the common starting point for all three attempts. The example then creates a branch for each attempt from that one checkpoint.
Use archil branches create to create the branches, naming the checkpoint to fork from. The loop below creates attempt-1, attempt-2 and attempt-3 from baseline. The final command shows the mount on the machine or sandbox running attempt 2. Branch selection happens through archil mount --branch. That makes the mounted workspace the attempt’s own branch.
# Once: mount the dataset disk and checkpoint it
sudo archil mount myorg/dataset /mnt/dataset --region aws-us-east-1
archil checkpoints create /mnt/dataset baseline
# One branch per attempt
for i in 1 2 3; do
archil branches --region aws-us-east-1 create dsk-0123456789abcdef attempt-$i \
--from-checkpoint baseline
done
# On the machine or sandbox running attempt 2
sudo archil mount myorg/dataset /mnt/attempt --region aws-us-east-1 --branch attempt-2
The mount paths have different roles in this example. /mnt/dataset is where the source disk is mounted when the checkpoint is created, while /mnt/attempt is where attempt 2 accesses its branch. The disk identifier in the branch-creation command names the disk being branched. The --from-checkpoint baseline argument selects its starting snapshot. These names connect the operations without making the three attempts share one writable mount.
Run each attempt in its own sandbox or machine with its corresponding branch mounted. Within that branch, the attempt can rewrite any file without affecting the others. For example, an attempt that cleans files can make its changes without imposing them on a sibling trying a different approach. Both still began from baseline.
Keeping the winning attempt
Score each branch the same way, for example by running the same validation script on each mounted branch. Starting from the same point gives the attempts the same inputs, while scoring them the same way puts their outputs through the same test. These address different parts of the comparison. The branch mechanism separates the attempts’ writes. Validation determines which result to keep.
The winning branch is already a complete, writable disk state. Keep mounting it to continue using that result, or checkpoint it and branch from it for the next round. A winner need not be a finish line. It can be the parent of another group of branches, each exploring a different next step.
The original checkpoint remains unchanged, so another set of attempts can start from it at any time. That leaves two distinct starting points available for different purposes. A new round can build on the selected result, while a fresh set can return to the original input.
Branches share the disk’s credentials, and anyone who can mount the disk can reach every branch. For untrusted attempts or separate tenants, give each its own disk. A branch keeps one attempt’s edits out of another branch; a separate disk also gives each attempt separate credentials.
Forking the whole sandbox instead of just the dataset
When an attempt needs installed tools and system state as well as data, fork the whole sandbox instead. The choice depends on what must be reproduced. If the tools and system state are part of the setup, the sandbox is the appropriate unit to fork.
Archil sandboxes keep their root filesystem on an Archil disk. The sandbox.fork() operation checkpoints that root disk and creates a branch from it, applying the same checkpoint-and-branch approach to the sandbox’s root filesystem. Rather than preparing only a dataset fork, the operation forks the disk that holds the sandbox’s root state.
Archil’s article on running parallel agent attempts from one starting state with copy-on-write branches covers forking a code workspace this way. For dataset-only work, the disk workflow above supplies the independent writable branches the attempts need. Fork what the attempt needs to change, without first duplicating everything it starts with.