# Archil for ML training: one warm dataset for every node, job, and cluster

Source: https://archil.com/solutions/ml-training

Keep your streaming loader and put one shared SSD cache over your training bucket under it: every node, job, and cluster reads warm data, nodes without local SSD included, and checkpoints are durable when fsync() returns.

ML training

# One warm dataset for every node, job, and cluster.

Keep your streaming loader. Archil puts your bucket behind one shared SSD cache that every node mounts as a disk, so back-to-back jobs and sweeps on any cluster read warm data, even on TPU VMs and other nodes with no local SSD to cache on.

[Talk to an integration specialist](https://cal.com/team/archil/demo)[Read the docs](https://docs.archil.com/getting-started/introduction)

Each node's loader fills its own cache from the bucket while it trains. The next job, or the next cluster, starts with empty caches and downloads again.

Runs where your accelerators are

-   _Amazon Web Services_
-   _Google Cloud_
-   _CoreWeave_
-   _Lambda_
-   _Nebius_

Mount a disk on any Linux node, with an IAM role on AWS or a disk token anywhere else. Public regions run on AWS and Google Cloud, and BYOC runs Archil in your own cloud or on bare metal. [How BYOC works](https://docs.archil.com/reference/byoc).

## What changes when the dataset is a disk.

A good streaming loader already starts training without a copy. Archil changes where the bytes live between jobs, and what a node needs to cache them.

01

### No local disk required

Accelerators with no Local SSD, like TPU VMs, have nowhere fast to put a loader's cache. Archil's client caches in memory, and a 4 KiB read that misses it comes back from the shared SSD cache in 0.5 ms at the median, so the node needs no disk of its own.

Before

no local SSD → cache on the boot disk

After

mount → read the shared cache

02

### Warm once for the whole sweep

The first job pulls each shard from the bucket into one shared cache, and every node, job, and cluster that mounts the disk reads it from there. Data stays cached for up to an hour after its last read, so back-to-back runs, sweeps, and restarts start warm.

Before

new job → empty caches → download again

After

new job → the cache is already warm

03

### The whole workspace, as files

Shards, raw files your code seeks into, tokenizers, eval sets, and checkpoints sit on one disk, as ordinary files and as plain objects in your bucket. Nothing is locked into one loader's format, and a checkpoint is durable when fsync() returns.

Before

samples in shards, everything else elsewhere

After

one disk for data, code, and checkpoints

04

### A local cache in every region

Training in more than one region? Put a disk in each one over the same bucket. Each disk pulls a shard across the globe once, then serves every node in its region from a cache next door, so no job streams samples from a bucket on another continent.

Before

far region → every node reads across the globe

After

far region → one pull, then a local cache

## What streaming loaders leave to you.

MosaicML Streaming, WebDataset, and Hugging Face streaming solved the staging problem: shards stream from the bucket while the GPUs train, with shuffling and sharding across nodes, and Streaming adds deterministic, elastic resumption. What they don't decide is where the bytes live.

### Every job starts cold

Each node's cache lives on that node's disk. A new cluster or a replaced node starts empty, and Streaming's own FAQ notes that with ephemeral storage you'll likely download shards again.

### No local SSD to cache on

TPU VMs have no Local SSD, and a multi-host slice's attached disks are read-only. Where GPU VMs have Local SSD, Google warns its data may be lost when the VM stops or is preempted.

### Samples, not files

Shard formats trade file access for sequential reads. MDS needs a conversion step, and Hugging Face's docs say a streamed dataset isn't for random access. WebDataset's tars stay readable by standard tools, but still as shards.

### Everything else needs a home

Checkpoints, tokenizers, eval sets, configs, and notebooks aren't samples. They get pushed to the bucket and pulled back down, or copied between clusters by hand.

## Already using a streaming loader? Keep it.

Put the shards on the Archil disk and give the loader the mount as a local path. Both loaders document this mode. The loader keeps its shuffling, sharding, and resumption, and the shared cache replaces each node's private copy.

streaming\_loader.py

[MosaicML Streaming](https://docs.mosaicml.com/projects/streaming/en/stable/)

```
from streaming import MDSWriter, StreamingDataset
from torch.utils.data import DataLoader
 
# Once: write MDS shards onto the disk, uncompressed,
# so every node reads them in place
with MDSWriter(out="/data/mds/train",
               columns={"tokens": "ndarray"}) as out:
    for sample in samples():
        out.write(sample)
 
# Every node, every job: local only, no remote and no
# cache_limit, so nothing is downloaded or evicted
dataset = StreamingDataset(local="/data/mds/train",
                           shuffle=True, batch_size=8)
loader = DataLoader(dataset, batch_size=8, num_workers=8)
```

wds\_loader.py

[WebDataset](https://github.com/webdataset/webdataset)

```
import webdataset as wds
 
# Tar shards on the disk, opened like local files.
# No cache_dir: the disk is already the shared cache.
urls = "/data/wds/train-{000000..000999}.tar"
 
dataset = (
    wds.WebDataset(urls, shardshuffle=1000,
                   nodesplitter=wds.split_by_node)
    .shuffle(1000)
    .decode("pil")
    .to_tuple("png", "json")
)
```

### Point local at the shards, with no remote

Streaming's local cache is per node: it tracks what's cached in each node's shared memory and evicts by deleting files. So don't aim several nodes' caches at one shared directory. Leave remote and cache\_limit unset, and read the shards in place.

[Streaming: shard retrieval](https://docs.mosaicml.com/projects/streaming/en/stable/dataset_configuration/shard_retrieval.html)

### Write the shards uncompressed

With compressed shards, Streaming decompresses each one into local before reading it, which on a shared disk means every node writing the same files. Uncompressed shards are read where they are.

### Write through the disk

Point MDSWriter or your tar writer at the mount, or at Archil's S3 API. Objects written straight to the bucket can take minutes to appear on the disk.

### DALI, Ray Data, and Hugging Face

DALI's readers take a directory; its docs suggest dont\_use\_mmap on network file systems. Ray Data reads shared files mounted at the same path on every node. Hugging Face streams local files without conversion.

[Ray Data: loading data](https://docs.ray.io/en/latest/data/loading-data.html)

## One disk in every training pod.

Point an Archil disk at the bucket that holds your data, and mount it ReadWriteMany into every training pod with the CSI driver. Loaders and training code keep opening and reading files.

[CSI driver docs](https://docs.archil.com/reference/csi-driver)

datasets.yaml

PersistentVolume and claim

```
# One Archil disk over your training bucket
apiVersion: v1
kind: PersistentVolume
metadata:
  name: datasets
spec:
  capacity:
    storage: 1000Gi  # Archil disks are elastic
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: csi.archil.com
    volumeHandle: "dsk-XXXXXXXXXXXX"
    volumeAttributes:
      region: "aws-us-east-1"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datasets
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1000Gi
  volumeName: datasets
```

trainer.yaml

Pod

```
# Every training pod mounts the same disk
apiVersion: v1
kind: Pod
metadata:
  name: trainer-0
spec:
  containers:
    - name: trainer
      image: "your-training-image"
      resources:
        limits:
          nvidia.com/gpu: 8
      volumeMounts:
        - name: datasets
          mountPath: /data
  volumes:
    - name: datasets
      persistentVolumeClaim:
        claimName: datasets
```

### Every pod reads and writes

ReadWriteMany mounts each pod in conditional mode, so any pod can read the dataset and write its checkpoints.

### Scale-out reads

Each client reads from many storage servers in parallel, the way Lustre scales out. Scale-out reads launched at 8 GiB/s per client.

### No read cap per disk

Reads per disk are unlimited. The highest one-minute average seen on one disk in production is 5.9 GB/s.

### Any S3-compatible bucket

Amazon S3, Google Cloud Storage, Cloudflare R2, or another S3-compatible store.

loader.py

Plain PyTorch, random access

```
import numpy as np
import torch
from torch.utils.data import DataLoader, Dataset
 
 
class TokenWindows(Dataset):
    """Random windows from one raw token file on the mount."""
 
    def __init__(self, path="/data/tokens/train.bin", ctx=2048):
        self.tokens = np.memmap(path, dtype=np.uint16, mode="r")
        self.ctx = ctx
 
    def __len__(self):
        return (len(self.tokens) - 1) // self.ctx
 
    def __getitem__(self, i):
        start = i * self.ctx
        window = self.tokens[start : start + self.ctx + 1]
        window = torch.from_numpy(window.astype(np.int64))
        return window[:-1], window[1:]
 
 
loader = DataLoader(TokenWindows(), batch_size=8,
                    shuffle=True, num_workers=8)
```

warm.sh

Outside Kubernetes

```
# A GPU or TPU VM outside Kubernetes: mount, and warm
# the tokenizer and validation set into this node's memory
archil mount <disk-name> /mnt/data --region aws-us-east-1 \
    --conditional \
    --max-cache-mb 65536 \
    --read-ahead 16777216 \
    --pre-fetch tokenizer,tokens/val
 
# Later, warm the files the next epoch starts with
archil prefetch /mnt/data tokens/train/part-0000 tokens/train/part-0001
```

### Warm what's next

archil prefetch, or --pre-fetch at mount, reads files and directories into a node's in-memory cache in the background, so the first read is already local.

### Tune read-ahead

The client reads up to 4 MiB ahead of sequential reads. Raise --read-ahead for large shards, and size --max-cache-mb for the working set.

[Caching docs](https://docs.archil.com/details/caching)

## Checkpoints that land once.

Write model checkpoints to the disk like any file. A write is durable on Archil once fsync() returns, stored redundantly across Availability Zones, and any node that mounts the disk can resume from it.

-   ### Snapshot the disk in 13 ms
    
    A disk checkpoint freezes the dataset, configs, and model checkpoints together, in 13 ms at the median, whatever the disk's size.
    
-   ### Branch every experiment in 33 ms
    
    Fork an independent, writable branch from any disk checkpoint in 33 ms at the median. Give each sweep or dataset version its own branch, and mount it on any node.
    
-   ### Evals and RL rollouts
    
    Fork a task environment per rollout, grade each run, and collect the graded trajectories on a disk.
    
    [Evals and RL environments guide](https://docs.archil.com/guides/training/rl-environments)

Checkpoints and branches run on Archil-managed disks in the AWS regions. Keep the raw corpus in your bucket, and put what you need to reproduce a run on a managed disk: the dataset manifests, configs, and model checkpoints. [Branches and checkpoints](https://docs.archil.com/concepts/branches-and-checkpoints). Writes reach up to 1 GB/s per disk today. If that limits how often you can save, tell us; most limits can be raised. [Limits](https://docs.archil.com/reference/limits).

save.py

Model checkpoints

```
import os
 
import torch
 
 
def save(state, step, root="/data/checkpoints/run-42"):
    path = f"{root}/step-{step:07d}.pt"
    with open(path + ".tmp", "wb") as f:
        torch.save(state, f)
        f.flush()
        os.fsync(f.fileno())  # durable on Archil when this returns
    os.rename(path + ".tmp", path)
```

sweep.sh

Disk checkpoints and branches

```
# Freeze the run, then fork two experiments from it
archil checkpoints create /data before-lr-sweep
 
archil branches --region aws-us-east-1 create dsk-XXXXXXXXXXXX \
    lr-3e-4 --from-checkpoint before-lr-sweep
archil branches --region aws-us-east-1 create dsk-XXXXXXXXXXXX \
    lr-1e-4 --from-checkpoint before-lr-sweep
 
# Each branch mounts anywhere, like any disk
archil mount <disk-name> /data --region aws-us-east-1 --branch lr-3e-4
```

## Bytes pulled from the bucket.

Set the shape of a sweep and compare what per-node loader caches download against filling one shared cache. Every input is an assumption you can change.

Your assumptions

Dataset size

10 TB

500 GB200 TB

Jobs over the dataset

8

1128

Nodes per job

16

1128

Each node reads, per job

Its partition

Its partitionAll of it

Later jobs with empty node caches

50%

0%100%

Breaks of over an hour between jobs

1

04

### A cache on every node

45 TBpulled from the bucket

Each cold job

10 TB

Jobs starting cold

4.5

Times the dataset

4.5×

A job starts cold on a new cluster, on replaced or preempted nodes, or where node storage doesn't outlive the job.

### One shared cache

20 TBpulled from the bucket

Each fill

10 TB

Times it fills

2

Times the dataset

2×

After each break of over an hour, the cache refills from the bucket as the next job reads.

A simplified model, not a benchmark. It counts bytes read out of the bucket, not what they cost, which depends on your provider and region, and not throughput or latency. It assumes every job reads the same samples, that node caches are large enough not to evict (a [cache\_limit set too low](https://docs.mosaicml.com/projects/streaming/en/stable/dataset_configuration/shard_retrieval.html) evicts and downloads shards again), and that the shared cache holds the working set: Archil's cache grows to fit it and keeps data for up to an hour after its last read, per its [published performance notes](https://docs.archil.com/details/performance). It leaves out that reads from the shared cache cross the network, where a node's own cache reads from its local disk.

## How the options compare.

Local NVMe is the fastest place to read data that's already there. A streaming loader removes the staging step and brings shuffling and resumption. A parallel file system serves tightly coupled jobs. Archil sits under a loader, for data that has to be in many places, on nodes with or without a disk to cache on.

How local NVMe staging, a streaming data loader, a parallel file system, FUSE over a bucket, and Archil compare for training
| Property | Local NVMe staging | Streaming data loader (MosaicML Streaming, WebDataset) | Parallel file system | FUSE over a bucket | Archil |
| --- | --- | --- | --- | --- | --- |
| Reads, once data is in place | Fastest: never leaves the box | From the node's own cache, after each shard downloads | High aggregate throughput for tightly coupled jobs | Each node's own reads from the bucket | A shared SSD cache, with scale-out reads per client |
| Before the first step | Copy the dataset to every node | Nothing: shards download as training reaches them | Load the dataset in, or link it to the bucket | Mount | Mount |
| Disk on each node | Room for the whole dataset | Room for its cache; cache\_limit bounds it | None | None, unless you turn on its file cache | None: the client caches in memory |
| Data format | Any files | Its own shards: MDS, or tar for WebDataset | Any files | Your objects, as files | Any files, kept as plain objects in your bucket |
| File semantics | Full POSIX | None: samples through the loader | Full POSIX | Partial: no locks or renames on Mountpoint | Standard POSIX operations |
| Shuffle and resume | Up to your loader | Built in: shard and buffer shuffles; Streaming adds elastic, deterministic resumption | Up to your loader | Up to your loader | Up to your loader, so keep the streaming one |
| A new job or cluster | Copies everything again | Reuses a cache only on the same node's disk | Usually its own file system, in one cloud | Downloads again, node by node | Mounts the same disk, warm for an hour after its last read |
| Checkpoints | Copied off by hand | Not its job: usually through the bucket | Shared within the file system | Through the bucket | Durable at fsync(), readable from any node |

### Local NVMe staging

Reads, once data is in place

Fastest: never leaves the box

Before the first step

Copy the dataset to every node

Disk on each node

Room for the whole dataset

Data format

Any files

File semantics

Full POSIX

Shuffle and resume

Up to your loader

A new job or cluster

Copies everything again

Checkpoints

Copied off by hand

### Streaming data loader (MosaicML Streaming, WebDataset)

Reads, once data is in place

From the node's own cache, after each shard downloads

Before the first step

Nothing: shards download as training reaches them

Disk on each node

Room for its cache; cache\_limit bounds it

Data format

Its own shards: MDS, or tar for WebDataset

File semantics

None: samples through the loader

Shuffle and resume

Built in: shard and buffer shuffles; Streaming adds elastic, deterministic resumption

A new job or cluster

Reuses a cache only on the same node's disk

Checkpoints

Not its job: usually through the bucket

### Parallel file system

Reads, once data is in place

High aggregate throughput for tightly coupled jobs

Before the first step

Load the dataset in, or link it to the bucket

Disk on each node

None

Data format

Any files

File semantics

Full POSIX

Shuffle and resume

Up to your loader

A new job or cluster

Usually its own file system, in one cloud

Checkpoints

Shared within the file system

### FUSE over a bucket

Reads, once data is in place

Each node's own reads from the bucket

Before the first step

Mount

Disk on each node

None, unless you turn on its file cache

Data format

Your objects, as files

File semantics

Partial: no locks or renames on Mountpoint

Shuffle and resume

Up to your loader

A new job or cluster

Downloads again, node by node

Checkpoints

Through the bucket

### Archil

Reads, once data is in place

A shared SSD cache, with scale-out reads per client

Before the first step

Mount

Disk on each node

None: the client caches in memory

Data format

Any files, kept as plain objects in your bucket

File semantics

Standard POSIX operations

Shuffle and resume

Up to your loader, so keep the streaming one

A new job or cluster

Mounts the same disk, warm for an hour after its last read

Checkpoints

Durable at fsync(), readable from any node

[How we measure performance](https://docs.archil.com/details/performance)[Data-intensive work beyond training](https://archil.com/use-cases/data-intensive)

## Bring your next training run.

[Talk to an integration specialist](https://cal.com/team/archil/demo)[Start building](https://console.archil.com)