Why S3 Performance Limits Matter — and How Archil Solves Them

AWS S3 is known for its massive scale, eleven nines of durability, and global availability. It’s the go-to storage layer for enterprises managing everything from logs and backups to massive machine learning datasets.

But S3 is an object store, not a file system—and that distinction matters.

S3 wasn’t built for low-latency, high-frequency access, or POSIX-style workloads. It lacks essential file system features, such as atomic renames, file locking, shared caching, and sub-millisecond response times. As a result, using S3 like a traditional file system leads to performance bottlenecks, inconsistent behavior, and engineering workarounds, especially as data volumes grow and concurrency demands rise.

What developers want is the durability of S3 with the speed and simplicity of a local file system without the operational overhead.

In this article, we’ll unpack the core performance limitations of S3 and explore how a solution like Archil bridges the gap for high-performance, cloud-native workloads.

S3 Explained: Capabilities and Misconceptions

To understand why treating S3 as a file system leads to bottlenecks, we must first examine its original design and intended use cases.

What S3 Is Designed For

Amazon S3 is a globally distributed object storage service optimized for massive scale and rock-solid durability. Its core capabilities include:

• Immutable Object Storage — Data is stored as immutable “objects” (up to 5 TiB each) in a flat namespace, identified by unique keys; updates require writing new objects.
• Massive Durability and Availability — Automatic multi-AZ replication and erasure coding deliver 99.99999999999% durability and 99.99% availability.
• Unlimited Scalability — Horizontally partitions by key prefixes across distributed nodes to handle trillions of objects and exabytes of data without manual tuning.
• API-Driven Access — Exposes a RESTful HTTP interface and SDKs instead of POSIX calls, enabling a global, language-agnostic integration.
• Strong Consistency — Guarantees read-after-write consistency for all PUTS and DELETES, ensuring immediate visibility of updates.

These capabilities make S3 an excellent system of record for write-once, read-many workloads. Examples include data-lake partitions, long-term backups, media archives, or large, immutable ML training datasets.

In these use cases, durability, availability, and virtually unlimited scale take precedence over sub-millisecond random access or full POSIX file-system semantics.

Common Misconceptions About S3

Despite its strengths, S3 is frequently misused under false assumptions, leading to brittle, underperforming systems. Some key misconceptions include:

1. “S3 is a POSIX File System” — S3 does not implement POSIX semantics. There is no 1) atomic rename, 2) file locking, 3) symbolic links, or 4) directory inodes. Applications relying on these primitives will break or exhibit undefined behavior. These mismatches force developers to introduce complex coordination layers, custom lock services, and copy-delete hacks, undermining performance and the correct use of the object store.
2. “FUSE Adapters Provide Native Semantics” — Tools like [s3fs](https://archil.com/article/efs-vs-s3#:~:text=One option is,replacement for EFS.) and Mountpoint for S3 expose a bucket over a mount point but cannot enforce genuine filesystem guarantees. They buffer or cache operations locally and replay them asynchronously, which can lead to timeouts, stale reads, write ordering issues, and caching bugs with concurrent access.
3. “Metadata Operations Are Inexpensive” — Operations like LIST, GET Bucket, and retrieving object metadata incur API call overhead, cost per request, and potential rate throttling. Unlike hierarchical structures in file systems, these calls in S3 traverse distributed indexes and are not optimized for high-frequency use.
4. “Throughput and IOPS Scale Linearly Without Effort” — S3 enforces per-prefix rate limits and per-connection throughput caps. Exceeding these without explicit prefix sharding and parallel streams results in throttling, increased latencies, and request failures.
5. “Latency is Negligible”—Typical object access latencies range quite drastically. For fine-grained, random-access workloads with small-file reads or high-frequency metadata operations, this latency is orders of magnitude higher than local or block storage.

These misconceptions illustrate why directly using S3 as a file system is an anti-pattern. In the next section, we will closely examine the architectural constraints of S3 that underlie these pitfalls.

Core S3 Performance Limitations

a. Prefix Partition Limits

S3 uses prefix-based partitioning to distribute objects and scale requests throughout its storage infrastructure. Each unique prefix in an S3 bucket is essentially a data shard, and for each prefix, S3 allocates storage and I/O capacity.

Given this architecture, AWS enforces hard per-prefix request limits. Currently, this is set to 3,500 PUT/POST/DELETE and 5,500 GET/HEAD operations per second/prefix. Consider an application that concentrates all its activity under a single prefix; it will quickly hit these ceilings and experience throttling, regardless of overall bucket capacity or the number of concurrent clients.

To avoid this bottleneck, developers must design key-naming strategies such as hashing or time-based prefixes to spread requests across partitions.

This adds a layer of complexity as developers must design custom logic for prefix distribution. For downstream read and list operations, multiple scans of pseudo-directories are needed to reconstruct a dataset.

b. Per-Connection Throughput Caps

Each TCP connection to S3 is limited to approximately 80 MiB/s, irrespective of the EC2 instance’s network capability or EBS throughput. This cap is enforced by S3’s connection policies, managing connection handoffs and buffer sizes to ensure fairness and stability across tenants. This design causes the following:

1. Single Stream Bottleneck: Even on a 100Gbps instance, a single [GET or PUT request cannot exceed ~80MiB/s](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html#:~:text=You can use the AWS,or 100 Gb/s NICs.). Large objects larger than 5 MiB are automatically uploaded or downloaded via multipart transfers, with each part adhering to the per-connection cap.
2. Client-Side Parallelism Required: To push beyond the bottleneck, applications must open multiple concurrent connections and manage them in parallel. For workloads needing 1 GiB/s, this typically entails orchestrating at least 13 simultaneous streams (~77 MiB/s per stream), along with thread pools, retry logic, and back-pressure handling.
3. Operational Complexity: Implementing efficient concurrent connections introduces significant engineering overhead:
   1. Synchronization of part writes and reads.
   2. Error Handling for individual stream failures.
   3. Load Balancing to avoid overloading prefixes.
   4. Monitoring to detect and recover from partial-throttle events.

Monitoring and observability are always a good idea. Especially when working with systems like S3 that can hit hidden performance ceilings. Platforms like Mezmo offer the ability to track latency trends, surface throttling events, and alert on anomalies, helping you stay ahead of bottlenecks before performance degradation.

These per-connection caps force developers to build custom multiplexing layers, introducing additional complexity and brittleness into their system architectures.

c. Latency and IOPS

S3 operations incur [10-100ms of round-trip delay](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html#:~:text=When you make,additional 4 seconds.) per request. This is orders of magnitude slower than local NVMe or even networked block storage (which delivers sub-millisecond latencies). This overhead stems from frequent HTTP API handling, authentication, and a multi-AZ replication pipeline. Frequent small-object reads or metadata calls can result in cumulative delays that slow random-access workloads significantly.

Unlike block storage, where you can provision and tune IOPS, S3’s capacity is bound by API rate limits and network performance. You cannot increase IOPS with configurations; you must distribute the  load across prefixes or establish parallel connections. High-IO workloads will often hit rate caps, resulting in inconsistent throttling or increased error rates.

d. Lack of POSIX Semantics

S3 is not a POSIX-compliant file system. It exposes a flat object namespace via HTTPS APIs, rather than  a hierarchical filesystem with primitives that applications expect. As a result, it omits critical POSIX features like:

• File Locking: There is no flock() or fcntl() support, so concurrent writers can’t coordinate writes or prevent race conditions.
• Atomic Renames: A POSIX rename() is not supported. Renaming requires a copy-and-delete sequence.
• Symbolic Links: S3 has no concept of inodes or links; each key is an isolated object.
• Random Writes: Objects are immutable, meaning you can’t modify a byte range in place. Updates must reupload whole objects or use multipart uploads as a workaround.

Applications that expect POSIX semantics, specifically data-processing tools, can behave unpredictably on S3.

Without point-in-time consistency, locks, or atomic directory operations, workflows encounter data corruption, dropped files, and subtle errors. This fundamental mismatch makes S3 unsuitable for workloads that rely on true filesystem behavior.

Real-World Impact on Workloads

These S3 constraints quickly become bottlenecks in practice.

ML training jobs that load thousands of small files suffer from high per-request latency and prefix throttling, causing idle compute resources. ETL pipelines must implement complex staging and custom lock services because S3 lacks atomic operations. Tools and research workflows that rely  on POSIX commands encounter race conditions and silent failures. When using spot or ephemeral instances, teams are forced to build local caching or synchronization layers, which adds  startup delays and risking stale data.

Why Archil Exists: Closing the Gap Between S3 and POSIX

It’s undeniable that developers rely on S3 for its scalability, durability, and seamless integrations across the cloud ecosystem. Its pay-as-you-go model, massive object store, and native support in data pipelines make it a default choice for modern infrastructure.

But as usage grows, so do the pain points: throttled prefixes, slow metadata operations, missing POSIX semantics, and connection throughput caps. These aren’t edge cases; they are daily obstacles for teams building high-performance ML pipelines, real-time apps, and complex ETL systems.

That’s why Archil exists: to bridge the gap between S3’s object storage model and the POSIX-compliant file systems developers expect.

What Archil Does: File System Performance, Backed by S3

Archil transforms your existing S3 buckets into high-performance, POSIX-compliant local file systems. It acts as a fully-managed, durable, high-speed caching layer between your compute environment and object storage, providing low-latency, consistent access to large datasets—without additional infrastructure provisioning or capacity planning.

Built for Performance: Low Latency, High Throughput, Zero Tuning

Applications mount Archil over an encrypted NFSv3 connection—no code changes are required. Archil then translates every file operation into the appropriate S3 API call, while a centralized cache tracks both data and metadata. The result is a seamless, high-performancefile system experience that is backed by S3, but without its usual limitations.

• Latency: Cached reads and writes return in sub-millisecond time. On a cache miss, Archil fetches the object from S3 in 10–30 ms, quicker than direct S3 access.
• Throughput & IOPS: Each file system delivers up to 10 Gbps and 10,000 IOPS by default, with higher tiers available on request.
• POSIX Compliance: Full support for file locks, renames, symbolic links, and random writes lets your applications behave exactly as they would on local storage, while you continue to benefit from S3’s scale, durability, and economics.

S3 Alone vs. S3 via Archil

When applications demand low latency, concurrent access, or full POSIX behavior, their limitations become more apparent. The table below compares using raw S3 directly versus layering Archil on top, highlighting where each approach fits best.

Raw S3 vs. Archil: Choosing the Best Storage Layer for POSIX, ML, and Real-Time Workloads

S3 remains a cornerstone of cloud storage and modern architecture. It excels as a scalable, cost-effective object store—ideal for static archives, logs, and cloud-native analytics that operate within the object paradigm.

But when your workflow demands file-system semantics and speed, those same strengths become friction. Per-prefix limits, connection caps, and the absence of POSIX features can slow development and force unnecessary workarounds.

Archil fills that gap. It adds high-performance caching, full POSIX compliance, and seamless integration, without requiring infrastructure, refactoring, or specialized tooling.

Use S3 alone when object storage is enough. Choose Archil when your cloud workloads demand low-latency access, traditional file behavior, and the scalability you already rely on in S3.

‍