# Archil for quantitative research: rerun any night exactly as production saw it

Source: https://archil.com/solutions/quant-research

Checkpoint production's whole disk after every nightly run and branch any night later to rerun it. Archil is a POSIX file system over your own bucket for tools that need one, shared by every machine in every zone, with read-only access for research and BYOC.

Quantitative research

# Rerun any night exactly as production saw it.

Archil checkpoints production's whole disk after every nightly run in about 13 ms. Weeks later, branch any night, mount it on any machine, and rerun it; production's disk never changes.

[Talk to an integration specialist](https://cal.com/team/archil/demo)[Read the docs](https://docs.archil.com/getting-started/introduction)

Runs where your research runs

-   _Amazon Web Services_
-   _Google Cloud_
-   _Kubernetes_
-   _Linux_

Any Linux machine or Kubernetes cluster. [How BYOC works](https://docs.archil.com/reference/byoc).

## Some research tools need a real file system.

A kdb+ historical database, compiled C++, and MATLAB research code open files by path. Object storage isn't a file system, so they usually land on a parallel one.

### kdb+ maps columns from a directory

A kdb+ historical database is a directory of column files that q memory-maps as queries touch them.

### On S3, kdb+ only reads

kdb+'s object-storage module reads a database from S3 but can't write one; it's copied up with the cloud's CLI.

### Lustre backups restore; they don't mount

Going back to last Tuesday means restoring a backup into a new file system.

Sources: [kdb+ memory mapping](https://kx.com/blog/memory-mapping-in-kdb/), [kdb+ object storage](https://code.kx.com/kdb-x/modules/objstor/overview.html), [FSx for Lustre deployment](https://docs.aws.amazon.com/fsx/latest/LustreGuide/using-fsx-lustre.html), [backups](https://docs.aws.amazon.com/fsx/latest/LustreGuide/using-backups-fsx.html), [restores](https://docs.aws.amazon.com/fsx/latest/LustreGuide/restoring-backups.html), [maintenance](https://docs.aws.amazon.com/fsx/latest/LustreGuide/maintenance-windows.html).

## A POSIX disk over the bucket you already have.

Files stay ordinary objects in your bucket; every machine mounts them as one shared disk.

### Programs open it like a local disk

Archil is a POSIX file system, so programs that memory-map or rename files open it the way they open a local disk.

### Every zone mounts the same disk

Workers in any availability zone of the region mount one disk, so a sweep runs wherever capacity is.

### Research reads production without touching it

Every credential covers one disk, and research's credential for production's disk is read-only.

### Publish a run with one rename

Tasks write into a hidden directory; one atomic rename publishes the whole run.

## Three disks: the lake, production, and research.

Point one disk at your lake's bucket; give production and research a managed disk each.

[How sharing works](https://docs.archil.com/concepts/sharing-disks)

### Who reads and writes what

Research reads the lake and production read-only, and writes only to its own disk.

sweep.py

Polars

```
import os
import sys
from datetime import date
 
import polars as pl
 
from research import grid, strategy  # your code
 
LAKE = "/mnt/lake"
RUNS = "/mnt/research/runs"
 
 
def run_task(run: str, task: int) -> None:
    """One point in the parameter grid. Every task
    reads the same bars; only the parameters change."""
    bars = (
        pl.scan_parquet(f"{LAKE}/bars/1m/**/*.parquet",
                        hive_partitioning=True)
        .filter(pl.col("date").is_between(
            date(2021, 1, 4), date(2026, 11, 4)))
    )
    pnl = strategy(bars, grid()[task]).collect()
 
    tmp = f"{RUNS}/.{run}.tmp"
    os.makedirs(tmp, exist_ok=True)
    pnl.write_parquet(f"{tmp}/part-{task:04d}.parquet")
 
 
def publish(run: str) -> None:
    """After the Job completes: one atomic rename."""
    os.rename(f"{RUNS}/.{run}.tmp", f"{RUNS}/{run}")
 
 
if __name__ == "__main__":  # python -m sweep run-42
    run_task(sys.argv[1],
             int(os.environ["JOB_COMPLETION_INDEX"]))
```

mount.sh

Three disks

```
# The lake: your bucket, read-only on every
# backtest worker and research box
sudo archil mount myorg/lake /mnt/lake \
    --region aws-us-east-1 --read-only
 
# Production's features, signals, and configs. The
# research role's credential for this disk is read-only
sudo archil mount myorg/prod /mnt/prod \
    --region aws-us-east-1 --read-only
 
# Research results: every task in a sweep writes here
sudo archil mount myorg/research /mnt/research \
    --region aws-us-east-1 --conditional
```

sweep.yaml

Indexed Job

```
# A 1,000-task grid search. When the Job completes:
#   python -c "import sweep; sweep.publish('run-42')"
apiVersion: batch/v1
kind: Job
metadata:
  name: sweep-run-42
spec:
  completionMode: Indexed  # sets JOB_COMPLETION_INDEX
  completions: 1000
  parallelism: 1000
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: backtest
          image: "your-research-image"
          command: ["python", "-m", "sweep", "run-42"]
          volumeMounts:
            - name: lake
              mountPath: /mnt/lake
              readOnly: true
            - name: research
              mountPath: /mnt/research
      volumes:
        - name: lake
          persistentVolumeClaim:
            claimName: lake      # RWX, csi.archil.com
        - name: research
          persistentVolumeClaim:
            claimName: research  # RWX, csi.archil.com
```

-   Up to 5,000 clients per disk and 8 GiB/s per client.
-   Files reach your bucket tens of seconds after their last write; Archil's S3 API sees them at once.

[CSI driver docs](https://docs.archil.com/reference/csi-driver)

## Checkpoint production after every run.

One command after the nightly run freezes every input, config, and output; a branch from it mounts anywhere.

[Branches and checkpoints](https://docs.archil.com/concepts/branches-and-checkpoints)

eod.sh

Archil CLI

```
# Production, after the nightly batch: freeze the disk
archil checkpoints create /mnt/prod eod-2026-11-04
 
# Weeks later: a writable branch from that night
archil branches --region aws-us-east-1 create dsk-XXXXXXXXXXXX \
    asof-2026-11-04 --from-checkpoint eod-2026-11-04
 
# Mount it anywhere and rerun against it
sudo archil mount myorg/prod /mnt/asof \
    --region aws-us-east-1 --branch asof-2026-11-04
```

replay.py

Rerun

```
import polars as pl
import yaml
 
from research import signal  # your code
 
ASOF = "/mnt/asof"  # production's disk, as of Nov 4
 
# Rerun November 4 against exactly the features and
# config production had that night
config = yaml.safe_load(open(f"{ASOF}/config/live.yaml"))
features = pl.scan_parquet(
    f"{ASOF}/features/date=2026-11-04/*.parquet")
rerun = signal(features, config).collect()
 
original = pl.read_parquet(f"{ASOF}/signals/2026-11-04.parquet")
print("same output:", rerun.equals(original))
```

Checkpoints run on Archil-managed disks in AWS regions, so production gets one; the lake in your bucket isn't checkpointed.

## A thousand backtests, one fetch.

The first read of a row group pulls it from your bucket; every other worker reads Archil's shared SSD cache, even at the same moment.

All 1,000 tasks start together, and each pulls its own copy of the bars.

## Where the data lives.

Choose how much of Archil runs in your account.

### BYOC: Archil in your cloud

Both layers run in your cloud account or on bare metal, operated over egress-only access.

[Bring your own cloud](https://docs.archil.com/reference/byoc)

### BYOB: files stay in your bucket

Each file is an ordinary object at its path, readable without Archil, and it stays if you leave.

[Data ownership](https://docs.archil.com/reference/data-ownership)

Active data stays in the disk's region. SOC 2 Type II, AES-256 at rest, TLS 1.2+ in transit. [Security and compliance](https://docs.archil.com/details/security).

## How the options compare.

A parallel file system gives POSIX tools a fast disk. Archil gives them one over your bucket, with checkpoints.

Ways to read Parquet for research, compared
| Property | Parquet on S3 (DuckDB, Polars) | Lakehouse tables (Delta, Iceberg, BigQuery) | Managed NFS (EFS, Filestore) | FSx for Lustre | Archil |
| --- | --- | --- | --- | --- | --- |
| Going back to a past night | Versions of each object | Time travel, within retention | Incremental backups | Restore a backup to a new file system | Whole-disk checkpoints, mounted as branches |
| Tools that need a file system | Through an SDK; kdb+ reads only | Through SQL engines | Full POSIX over NFS | Full POSIX | POSIX, over your bucket |
| 1,000 tasks, same bars | Each task fetches its own | The engine's own cache | Served once copied in | Loaded on first access | Fetched once, then shared cache |
| Publishing a run | No rename; a marker file | Atomic table commits | Atomic rename | Atomic rename | Atomic rename; bucket follows in seconds |
| Read-only access, experiments | Read-only IAM; copies | Table grants; Iceberg branches | Read-only IAM; copies | POSIX permissions; copies | Read-only credentials; branches |

### Parquet on S3 (DuckDB, Polars)

Going back to a past night

Versions of each object

Tools that need a file system

Through an SDK; kdb+ reads only

1,000 tasks, same bars

Each task fetches its own

Publishing a run

No rename; a marker file

Read-only access, experiments

Read-only IAM; copies

### Lakehouse tables (Delta, Iceberg, BigQuery)

Going back to a past night

Time travel, within retention

Tools that need a file system

Through SQL engines

1,000 tasks, same bars

The engine's own cache

Publishing a run

Atomic table commits

Read-only access, experiments

Table grants; Iceberg branches

### Managed NFS (EFS, Filestore)

Going back to a past night

Incremental backups

Tools that need a file system

Full POSIX over NFS

1,000 tasks, same bars

Served once copied in

Publishing a run

Atomic rename

Read-only access, experiments

Read-only IAM; copies

### FSx for Lustre

Going back to a past night

Restore a backup to a new file system

Tools that need a file system

Full POSIX

1,000 tasks, same bars

Loaded on first access

Publishing a run

Atomic rename

Read-only access, experiments

POSIX permissions; copies

### Archil

Going back to a past night

Whole-disk checkpoints, mounted as branches

Tools that need a file system

POSIX, over your bucket

1,000 tasks, same bars

Fetched once, then shared cache

Publishing a run

Atomic rename; bucket follows in seconds

Read-only access, experiments

Read-only credentials; branches

[How performance is measured](https://docs.archil.com/details/performance)[Data-intensive work beyond research](https://archil.com/use-cases/data-intensive)

## Branch last night and rerun it.

[Talk to an integration specialist](https://cal.com/team/archil/demo)[Start building](https://console.archil.com)