Life sciences

One disk over your bucket, mounted on every node.

Archil turns your bucket into one disk that every node mounts, with a shared cache in front. Tasks open files in place, and the bucket serves each file once.

Each task copies the reference files into its own scratch, and the next task copies them again.

Mounts where your pipelines run

  • Amazon Web Services
  • Google Cloud
  • Kubernetes
  • Linux

Slurm nodes and EKS or GKE pods mount one disk, on AWS, Google Cloud, or BYOC in your account. How BYOC works.

Why pipelines still copy.

Most options still copy shared files to every node, or need a disk provisioned to hold them.

Copies in, copies out

Nextflow on AWS Batch, Cromwell, Snakemake storage plugins, and HealthOmics copy inputs in before a task runs.

A cache per node, or per process

Mountpoint, Cloud Storage FUSE, and the FUSE layer sold for Nextflow cache per node, so each new node pulls the files again.

Capacity to size up front

FSx for Lustre needs capacity set before the run, in 1.2 or 2.4 TiB steps, in one Availability Zone.

Files that aren't quite files

Tools that need the crossed-out operations still need a scratch disk beside Mountpoint.

What a shared disk changes.

One disk over your bucket, at the same path on every node.

Many readers, one fetch

Tasks that open a file mid-fetch wait on that fetch; later ones, on any node, read the shared cache until it has sat an hour unread.

No stage-in, no stage-out

Outputs are durable when fsync() returns, and reach your bucket at the same key tens of seconds later.

Run where the capacity is

Nodes in any zone of the region mount the disk, so jobs run wherever instances are free.

The bucket stays the source of truth

Direct writes to the bucket appear within minutes; uploads through Archil's S3 API appear on return.

Tools that expect a shared file system.

Nextflow, Snakemake, and cryoSPARC expect a file system every node sees at one path. An Archil disk is one, over your bucket.

Nextflow · Snakemake

Genomics pipelines

The work directory gets its own Archil-managed disk, so intermediates stay out of your bucket.

cryoSPARC · RELION

Imaging

Every worker sees one absolute path; repeat reads within the hour come from the shared cache.

Python · R · bash

Research agents

Each tool call runs in a fresh microVM beside the data: the cohort read-only, the session read-write.

  • 2 vCPU, 4 GiB, and up to 5 minutes per call; sandboxes run longer, with up to 32 vCPU and 64 GiB.
Serverless execution docs
agent_tool.py
Serverless execution
import archil
from archil import ExecMountSpec
 
# Exec allows 5 minutes per command; the client's default
# request timeout is 30 seconds, so raise it.
client = archil.Archil(timeout=330)
 
# The cohort mounts read-only; each session has its own disk.
COHORT = ExecMountSpec("dsk-XXXXXXXXXXXX", read_only=True)
 
 
def bash(session_disk: str, command: str):
"""The agent's bash tool: one fresh microVM per call."""
res = client.exec(
disks={"cohort": COHORT, "session": session_disk},
command=command,
)
return res.stdout, res.stderr, res.exit_code
 
 
# The report stays on the session disk for the next call,
# and your app can read it over the S3 API.
bash("dsk-YYYYYYYYYYYY",
"python3 session/survival.py "
"--expr cohort/expression/tpm.tsv "
"--clinical cohort/clinical.tsv "
"--out session/reports/survival.md")

Two disks and a config change.

One disk over your bucket, a second for the work directory, both mounted on every node.

Connect your bucket
disk.tf
Terraform, over your bucket
# The disk over the bucket your data already
# lives in. Nothing is copied or converted.
resource "archil_disk" "genomics" {
name = "genomics"
 
mount {
type = "s3"
bucket_name = "your-genomics-bucket"
}
}
 
# Nextflow's work directory gets its own
# Archil-managed disk, so intermediates never
# land in your bucket.
resource "archil_disk" "work" {
name = "nextflow-work"
}
 
# With no access keys, your bucket policy grants
# Archil's role access, scoped to this one disk
# by its session ID.
output "archil_session_id" {
value = archil_disk.genomics.mount[0].session_id
}
mount.sh
Every compute node
# At boot on every Slurm compute node
curl -s https://archil.com/install | sh
 
# On AWS, the node's IAM role is a disk user: no token.
# --conditional lets every node write at once.
sudo mkdir -p /mnt/archil /mnt/work
sudo archil mount genomics /mnt/archil \
--region aws-us-east-1 --conditional
sudo archil mount nextflow-work /mnt/work \
--region aws-us-east-1 --conditional
 
# close() sends a file's writes to the disk, and creates
# and renames land before they return, so another node
# that opens the file next reads them. Listings cache
# for five minutes by default; refresh the work
# directory's in seconds.
archil set-cache-expiry /mnt/work --readdir-expiry 5
nextflow.config
// nf-core/sarek on Kubernetes, EKS included. Both
// claims are ReadWriteMany volumes on the Archil CSI
// driver. (Nextflow's AWS Batch executor stages
// inputs through S3, so use k8s or Slurm.)
process.executor = 'k8s'
 
k8s {
storageClaimName = 'nextflow-work'
storageMountPath = '/mnt/work'
}
workDir = '/mnt/work'
 
process.pod = [volumeClaim: 'genomics', mountPath: '/mnt/archil']
 
// Point at prebuilt index files, so the pipeline
// reads them in place instead of building them.
params {
igenomes_ignore = true
fasta = '/mnt/archil/references/GRCh38/GRCh38.fa'
bwamem2 = '/mnt/archil/references/GRCh38/bwamem2/'
input = '/mnt/archil/runs/2026-11/samplesheet.csv'
outdir = '/mnt/archil/results/2026-11'
}
# Snakemake on Slurm, with the project on the disk.
# The nodes all see the same paths: no storage plugin.
cd /mnt/archil/projects/wgs-2026-11
snakemake --executor slurm --jobs 200
  • No read cap per disk; up to 8 GiB/s per client.
  • No local disk needed: the cache is RAM, 2 GiB by default.

Where the data lives.

Data stays in your bucket and region, and Archil can run inside your account.

Security and compliance

Your bucket, your region, your account

Files stay ordinary objects in your bucket and region; with BYOC, no PHI leaves your account.

Access per disk

An IAM role or disk token opens one disk, and POSIX permissions apply on each mount.

HIPAA, SOC 2, and encryption

HIPAA workloads, with a BAA for Enterprise customers, and ongoing SOC 2 Type II audits.

How the options compare.

Staging is hard to beat on one big node. On a fleet, every task copies.

Stage in to task scratch

Before a task computes
Copy inputs in
Shared inputs, across nodes
Copied by every task
File semantics
Full POSIX, locally
Shared work directory
No: S3 on AWS Batch
Capacity
Sized per task
Replication
Bucket's; scratch is ephemeral

FUSE over the bucket

Before a task computes
Nothing: reads on demand
Shared inputs, across nodes
Cached per node
File semantics
Varies; Mountpoint can't modify files
Shared work directory
Only with a Nextflow FUSE layer
Capacity
Local cache disk per node
Replication
Bucket's; caches are ephemeral

FSx for Lustre linked to S3

Before a task computes
Nothing: loads on first access
Shared inputs, across nodes
Loaded once, shared
File semantics
Full POSIX
Shared work directory
Yes
Capacity
Set up front, in TiB steps
Replication
One Availability Zone

Amazon EFS or S3 Files

Before a task computes
EFS: load first; S3 Files: nothing
Shared inputs, across nodes
Shared
File semantics
NFS; S3 Files lacks hard links
Shared work directory
Yes
Capacity
Elastic
Replication
Across zones

Archil

Before a task computes
Nothing: reads in place
Shared inputs, across nodes
One fetch, even concurrently
File semantics
POSIX, renames, random writes
Shared work directory
Yes, ReadWriteMany
Capacity
Elastic; no local disk
Replication
Across zones, then the bucket