Quantitative research

Rerun any night exactly as production saw it.

Archil checkpoints production's whole disk after every nightly run in about 13 ms. Weeks later, branch any night, mount it on any machine, and rerun it; production's disk never changes.

Runs where your research runs

  • Amazon Web Services
  • Google Cloud
  • Kubernetes
  • Linux

Any Linux machine or Kubernetes cluster. How BYOC works.

Some research tools need a real file system.

A kdb+ historical database, compiled C++, and MATLAB research code open files by path. Object storage isn't a file system, so they usually land on a parallel one.

kdb+ maps columns from a directory

A kdb+ historical database is a directory of column files that q memory-maps as queries touch them.

On S3, kdb+ only reads

kdb+'s object-storage module reads a database from S3 but can't write one; it's copied up with the cloud's CLI.

Lustre backups restore; they don't mount

Going back to last Tuesday means restoring a backup into a new file system.

Sources: kdb+ memory mapping, kdb+ object storage, FSx for Lustre deployment, backups, restores, maintenance.

A POSIX disk over the bucket you already have.

Files stay ordinary objects in your bucket; every machine mounts them as one shared disk.

Programs open it like a local disk

Archil is a POSIX file system, so programs that memory-map or rename files open it the way they open a local disk.

Every zone mounts the same disk

Workers in any availability zone of the region mount one disk, so a sweep runs wherever capacity is.

Research reads production without touching it

Every credential covers one disk, and research's credential for production's disk is read-only.

Publish a run with one rename

Tasks write into a hidden directory; one atomic rename publishes the whole run.

Three disks: the lake, production, and research.

Point one disk at your lake's bucket; give production and research a managed disk each.

How sharing works

Who reads and writes what

Research reads the lake and production read-only, and writes only to its own disk.

sweep.py
Polars
import os
import sys
from datetime import date
 
import polars as pl
 
from research import grid, strategy # your code
 
LAKE = "/mnt/lake"
RUNS = "/mnt/research/runs"
 
 
def run_task(run: str, task: int) -> None:
"""One point in the parameter grid. Every task
reads the same bars; only the parameters change."""
bars = (
pl.scan_parquet(f"{LAKE}/bars/1m/**/*.parquet",
hive_partitioning=True)
.filter(pl.col("date").is_between(
date(2021, 1, 4), date(2026, 11, 4)))
)
pnl = strategy(bars, grid()[task]).collect()
 
tmp = f"{RUNS}/.{run}.tmp"
os.makedirs(tmp, exist_ok=True)
pnl.write_parquet(f"{tmp}/part-{task:04d}.parquet")
 
 
def publish(run: str) -> None:
"""After the Job completes: one atomic rename."""
os.rename(f"{RUNS}/.{run}.tmp", f"{RUNS}/{run}")
 
 
if __name__ == "__main__": # python -m sweep run-42
run_task(sys.argv[1],
int(os.environ["JOB_COMPLETION_INDEX"]))
mount.sh
Three disks
# The lake: your bucket, read-only on every
# backtest worker and research box
sudo archil mount myorg/lake /mnt/lake \
--region aws-us-east-1 --read-only
 
# Production's features, signals, and configs. The
# research role's credential for this disk is read-only
sudo archil mount myorg/prod /mnt/prod \
--region aws-us-east-1 --read-only
 
# Research results: every task in a sweep writes here
sudo archil mount myorg/research /mnt/research \
--region aws-us-east-1 --conditional
sweep.yaml
Indexed Job
# A 1,000-task grid search. When the Job completes:
# python -c "import sweep; sweep.publish('run-42')"
apiVersion: batch/v1
kind: Job
metadata:
name: sweep-run-42
spec:
completionMode: Indexed # sets JOB_COMPLETION_INDEX
completions: 1000
parallelism: 1000
template:
spec:
restartPolicy: Never
containers:
- name: backtest
image: "your-research-image"
command: ["python", "-m", "sweep", "run-42"]
volumeMounts:
- name: lake
mountPath: /mnt/lake
readOnly: true
- name: research
mountPath: /mnt/research
volumes:
- name: lake
persistentVolumeClaim:
claimName: lake # RWX, csi.archil.com
- name: research
persistentVolumeClaim:
claimName: research # RWX, csi.archil.com
  • Up to 5,000 clients per disk and 8 GiB/s per client.
  • Files reach your bucket tens of seconds after their last write; Archil's S3 API sees them at once.
CSI driver docs

Checkpoint production after every run.

One command after the nightly run freezes every input, config, and output; a branch from it mounts anywhere.

Branches and checkpoints
eod.sh
Archil CLI
# Production, after the nightly batch: freeze the disk
archil checkpoints create /mnt/prod eod-2026-11-04
 
# Weeks later: a writable branch from that night
archil branches --region aws-us-east-1 create dsk-XXXXXXXXXXXX \
asof-2026-11-04 --from-checkpoint eod-2026-11-04
 
# Mount it anywhere and rerun against it
sudo archil mount myorg/prod /mnt/asof \
--region aws-us-east-1 --branch asof-2026-11-04
replay.py
Rerun
import polars as pl
import yaml
 
from research import signal # your code
 
ASOF = "/mnt/asof" # production's disk, as of Nov 4
 
# Rerun November 4 against exactly the features and
# config production had that night
config = yaml.safe_load(open(f"{ASOF}/config/live.yaml"))
features = pl.scan_parquet(
f"{ASOF}/features/date=2026-11-04/*.parquet")
rerun = signal(features, config).collect()
 
original = pl.read_parquet(f"{ASOF}/signals/2026-11-04.parquet")
print("same output:", rerun.equals(original))

Checkpoints run on Archil-managed disks in AWS regions, so production gets one; the lake in your bucket isn't checkpointed.

A thousand backtests, one fetch.

The first read of a row group pulls it from your bucket; every other worker reads Archil's shared SSD cache, even at the same moment.

All 1,000 tasks start together, and each pulls its own copy of the bars.

Where the data lives.

Choose how much of Archil runs in your account.

BYOC: Archil in your cloud

Both layers run in your cloud account or on bare metal, operated over egress-only access.

Bring your own cloud

BYOB: files stay in your bucket

Each file is an ordinary object at its path, readable without Archil, and it stays if you leave.

Data ownership

Active data stays in the disk's region. SOC 2 Type II, AES-256 at rest, TLS 1.2+ in transit. Security and compliance.

How the options compare.

A parallel file system gives POSIX tools a fast disk. Archil gives them one over your bucket, with checkpoints.

Parquet on S3 (DuckDB, Polars)

Going back to a past night
Versions of each object
Tools that need a file system
Through an SDK; kdb+ reads only
1,000 tasks, same bars
Each task fetches its own
Publishing a run
No rename; a marker file
Read-only access, experiments
Read-only IAM; copies

Lakehouse tables (Delta, Iceberg, BigQuery)

Going back to a past night
Time travel, within retention
Tools that need a file system
Through SQL engines
1,000 tasks, same bars
The engine's own cache
Publishing a run
Atomic table commits
Read-only access, experiments
Table grants; Iceberg branches

Managed NFS (EFS, Filestore)

Going back to a past night
Incremental backups
Tools that need a file system
Full POSIX over NFS
1,000 tasks, same bars
Served once copied in
Publishing a run
Atomic rename
Read-only access, experiments
Read-only IAM; copies

FSx for Lustre

Going back to a past night
Restore a backup to a new file system
Tools that need a file system
Full POSIX
1,000 tasks, same bars
Loaded on first access
Publishing a run
Atomic rename
Read-only access, experiments
POSIX permissions; copies

Archil

Going back to a past night
Whole-disk checkpoints, mounted as branches
Tools that need a file system
POSIX, over your bucket
1,000 tasks, same bars
Fetched once, then shared cache
Publishing a run
Atomic rename; bucket follows in seconds
Read-only access, experiments
Read-only credentials; branches