Quantitative research
Rerun any night exactly as production saw it.
Archil checkpoints production's whole disk after every nightly run in about 13 ms. Weeks later, branch any night, mount it on any machine, and rerun it; production's disk never changes.
Runs where your research runs
Any Linux machine or Kubernetes cluster. How BYOC works.
Some research tools need a real file system.
A kdb+ historical database, compiled C++, and MATLAB research code open files by path. Object storage isn't a file system, so they usually land on a parallel one.
kdb+ maps columns from a directory
A kdb+ historical database is a directory of column files that q memory-maps as queries touch them.
On S3, kdb+ only reads
kdb+'s object-storage module reads a database from S3 but can't write one; it's copied up with the cloud's CLI.
Lustre backups restore; they don't mount
Going back to last Tuesday means restoring a backup into a new file system.
Sources: kdb+ memory mapping, kdb+ object storage, FSx for Lustre deployment, backups, restores, maintenance.
A POSIX disk over the bucket you already have.
Files stay ordinary objects in your bucket; every machine mounts them as one shared disk.
Programs open it like a local disk
Archil is a POSIX file system, so programs that memory-map or rename files open it the way they open a local disk.
Every zone mounts the same disk
Workers in any availability zone of the region mount one disk, so a sweep runs wherever capacity is.
Research reads production without touching it
Every credential covers one disk, and research's credential for production's disk is read-only.
Publish a run with one rename
Tasks write into a hidden directory; one atomic rename publishes the whole run.
Three disks: the lake, production, and research.
Point one disk at your lake's bucket; give production and research a managed disk each.
How sharing worksWho reads and writes what
Research reads the lake and production read-only, and writes only to its own disk.
import osimport sysfrom datetime import date import polars as pl from research import grid, strategy # your code LAKE = "/mnt/lake"RUNS = "/mnt/research/runs" def run_task(run: str, task: int) -> None: """One point in the parameter grid. Every task reads the same bars; only the parameters change.""" bars = ( pl.scan_parquet(f"{LAKE}/bars/1m/**/*.parquet", hive_partitioning=True) .filter(pl.col("date").is_between( date(2021, 1, 4), date(2026, 11, 4))) ) pnl = strategy(bars, grid()[task]).collect() tmp = f"{RUNS}/.{run}.tmp" os.makedirs(tmp, exist_ok=True) pnl.write_parquet(f"{tmp}/part-{task:04d}.parquet") def publish(run: str) -> None: """After the Job completes: one atomic rename.""" os.rename(f"{RUNS}/.{run}.tmp", f"{RUNS}/{run}") if __name__ == "__main__": # python -m sweep run-42 run_task(sys.argv[1], int(os.environ["JOB_COMPLETION_INDEX"]))# The lake: your bucket, read-only on every# backtest worker and research boxsudo archil mount myorg/lake /mnt/lake \ --region aws-us-east-1 --read-only # Production's features, signals, and configs. The# research role's credential for this disk is read-onlysudo archil mount myorg/prod /mnt/prod \ --region aws-us-east-1 --read-only # Research results: every task in a sweep writes heresudo archil mount myorg/research /mnt/research \ --region aws-us-east-1 --conditional# A 1,000-task grid search. When the Job completes:# python -c "import sweep; sweep.publish('run-42')"apiVersion: batch/v1kind: Jobmetadata: name: sweep-run-42spec: completionMode: Indexed # sets JOB_COMPLETION_INDEX completions: 1000 parallelism: 1000 template: spec: restartPolicy: Never containers: - name: backtest image: "your-research-image" command: ["python", "-m", "sweep", "run-42"] volumeMounts: - name: lake mountPath: /mnt/lake readOnly: true - name: research mountPath: /mnt/research volumes: - name: lake persistentVolumeClaim: claimName: lake # RWX, csi.archil.com - name: research persistentVolumeClaim: claimName: research # RWX, csi.archil.com- Up to 5,000 clients per disk and 8 GiB/s per client.
- Files reach your bucket tens of seconds after their last write; Archil's S3 API sees them at once.
Checkpoint production after every run.
One command after the nightly run freezes every input, config, and output; a branch from it mounts anywhere.
Branches and checkpoints# Production, after the nightly batch: freeze the diskarchil checkpoints create /mnt/prod eod-2026-11-04 # Weeks later: a writable branch from that nightarchil branches --region aws-us-east-1 create dsk-XXXXXXXXXXXX \ asof-2026-11-04 --from-checkpoint eod-2026-11-04 # Mount it anywhere and rerun against itsudo archil mount myorg/prod /mnt/asof \ --region aws-us-east-1 --branch asof-2026-11-04import polars as plimport yaml from research import signal # your code ASOF = "/mnt/asof" # production's disk, as of Nov 4 # Rerun November 4 against exactly the features and# config production had that nightconfig = yaml.safe_load(open(f"{ASOF}/config/live.yaml"))features = pl.scan_parquet( f"{ASOF}/features/date=2026-11-04/*.parquet")rerun = signal(features, config).collect() original = pl.read_parquet(f"{ASOF}/signals/2026-11-04.parquet")print("same output:", rerun.equals(original))Checkpoints run on Archil-managed disks in AWS regions, so production gets one; the lake in your bucket isn't checkpointed.
A thousand backtests, one fetch.
The first read of a row group pulls it from your bucket; every other worker reads Archil's shared SSD cache, even at the same moment.
All 1,000 tasks start together, and each pulls its own copy of the bars.
Where the data lives.
Choose how much of Archil runs in your account.
BYOC: Archil in your cloud
Both layers run in your cloud account or on bare metal, operated over egress-only access.
Bring your own cloudBYOB: files stay in your bucket
Each file is an ordinary object at its path, readable without Archil, and it stays if you leave.
Data ownershipActive data stays in the disk's region. SOC 2 Type II, AES-256 at rest, TLS 1.2+ in transit. Security and compliance.
How the options compare.
A parallel file system gives POSIX tools a fast disk. Archil gives them one over your bucket, with checkpoints.
| Property | Parquet on S3 (DuckDB, Polars) | Lakehouse tables (Delta, Iceberg, BigQuery) | Managed NFS (EFS, Filestore) | FSx for Lustre | Archil |
|---|---|---|---|---|---|
| Going back to a past night | Versions of each object | Time travel, within retention | Incremental backups | Restore a backup to a new file system | Whole-disk checkpoints, mounted as branches |
| Tools that need a file system | Through an SDK; kdb+ reads only | Through SQL engines | Full POSIX over NFS | Full POSIX | POSIX, over your bucket |
| 1,000 tasks, same bars | Each task fetches its own | The engine's own cache | Served once copied in | Loaded on first access | Fetched once, then shared cache |
| Publishing a run | No rename; a marker file | Atomic table commits | Atomic rename | Atomic rename | Atomic rename; bucket follows in seconds |
| Read-only access, experiments | Read-only IAM; copies | Table grants; Iceberg branches | Read-only IAM; copies | POSIX permissions; copies | Read-only credentials; branches |
Parquet on S3 (DuckDB, Polars)
- Going back to a past night
- Versions of each object
- Tools that need a file system
- Through an SDK; kdb+ reads only
- 1,000 tasks, same bars
- Each task fetches its own
- Publishing a run
- No rename; a marker file
- Read-only access, experiments
- Read-only IAM; copies
Lakehouse tables (Delta, Iceberg, BigQuery)
- Going back to a past night
- Time travel, within retention
- Tools that need a file system
- Through SQL engines
- 1,000 tasks, same bars
- The engine's own cache
- Publishing a run
- Atomic table commits
- Read-only access, experiments
- Table grants; Iceberg branches
Managed NFS (EFS, Filestore)
- Going back to a past night
- Incremental backups
- Tools that need a file system
- Full POSIX over NFS
- 1,000 tasks, same bars
- Served once copied in
- Publishing a run
- Atomic rename
- Read-only access, experiments
- Read-only IAM; copies
FSx for Lustre
- Going back to a past night
- Restore a backup to a new file system
- Tools that need a file system
- Full POSIX
- 1,000 tasks, same bars
- Loaded on first access
- Publishing a run
- Atomic rename
- Read-only access, experiments
- POSIX permissions; copies
Archil
- Going back to a past night
- Whole-disk checkpoints, mounted as branches
- Tools that need a file system
- POSIX, over your bucket
- 1,000 tasks, same bars
- Fetched once, then shared cache
- Publishing a run
- Atomic rename; bucket follows in seconds
- Read-only access, experiments
- Read-only credentials; branches