If your application requires persistent local storage, one that does not get deleted when a container is stopped, you can not go Serverless/Fargate. That is not even a tradeoff, that is a limit.
Local EBS
A persistent local storage is a hard drive, or Elastic Block Store (EBS) at AWS. Is it the fastest, cheapest and most reliable. Alas it can be attached to only one server at a time, which contradicts one of the Orchestrator’s main purposes, to bin back.
There are applications/databases that depend on a file system (EBS). if the Orchestrator were to move it from an old underlying server to another new server, the EBS would not be accessible from the new server as it is still attached to the old server.
You can instruct the Orchestrator not to move it anywhere, to make it stay put on a single server for its entire lifetime, but there is a small tradeoff to know of. It would incur some additional costs due to under utilization as it would somewhat affect the Orchestrator’s bin backing process. That is considered the practice in this case.
I’m putting in another solution I’ve worked on and dropped for future reference. You can move an EBS from one server to another, which has been done for years when a server (or it’s OS) has been decommissioned for some reason. But having the Orchestrator automatically move EBS between servers is a whole other thing.
I’ve once (2017) codec a PoC that does exactly. There also used to be a docker plugin called Cloudstor that did something similar. But as of the time of writing this, I can barely find any references to it. It may have been dropped.
While EBS it is being reattached, it is not available to the application. It would also require so applicative changes, as the data on the EBS is no longer fresh. Lastly, EBS can not move across availability zones that easily and quickly.
Remote EFS
While I was trying to resolve the EBS issue (2017), there was another option available, a shared network Elastic File System (EFS). Back then, EFS was lacking a lot of EBS’s capabilities such as backup and restore. As I’m reviewing it again in 2020, EFS has caught up speed with EBS’s capabilites and using it as a shared persistent storage is considered a good practice. There are still caveats and tradeoffs, these will be latency and costs.
EBS throughput ranges between 250MiB/s to 1GiB/s, with barely any respect to volume size and you’d be paying $0.1 a month per GB.
EFS would cost you three times that, $0.3 a month per GB for a lower throughput. The upside is that It is concurrently available to all EC2 instances and all Containers instances whether on ECS or Fargate. That’s an additional $20 for 100GB and $200 1TB a month.
EFS throughput grows as the file system size grows. The higher the throughput you need, the more you’d pay for unused storage. For example, a 10GiB file system would have a throughput of mere 0.5MiB/S that would cost $3 a month. A 200MiB/s throughput would require a 4TB file system that would cost $1,200 a month. EBS would have cost 400$ with 5 times the throughput.
You would pay dearly if your application’s throughput would turn up to be high, or would one day scale to that throughput. EFS is designed and dedicated to relatively infrequent access. It is not intended for continuously accessed / high throughput databases, although it technically can.
Let’s take ElasticSearch as a baseline for two hypothetical use cases. The first which resembles Silo’s Event Analysis, is one that the freshness of the database can be measured in minutes and stores data for only a month. Query rate is fairly low as the application is being used by only a few employees that issue a query once in a few tens of seconds. If we were not to use a fully managed service, EFS might have been a good storage solution, as the required throughput is relatively low (the sharding strategy allowed that) and latency is not much of an issue.
On the contrary, Silo’s Food Intention Service which was a customer facing application with minimal latency required and with a high query rate, EFS would not be a wise path to follow at all.
Remote S3
Amazon’s Scalable Storage Solution (S3) is not a managed file system but rather a managed object/file management system. You can not run Redis/ElasticSearch/MySQL or any other application that needs continuous access / open file / file descriptor to its files. I’m mentioning it here in this context as it is a shared file solution which can be used by multiple applications instances. All files can be accessed and handled via HTTP.
If your application is a simple one that only requires storing and getting a JSON file per user, or any other file type and sharded key, maybe consider not storing it in a database at all. Querying file content is somewhat possible as well. That would eliminate the need for a persistent file storage in the first place. You’d be not solving a problem that no longer exists.
Vertical Scaling of Read Replicas
I’ve recently seen a new scaling/distributed strategy for databases that is interesting in the context of Containers and persistent storage. [A topic I have some practical experience with and some theoretical one]
There is a scaling strategy where your database instances are split into two roles: write replicas and read replicas. As each one requires a different throughput, each one can be scaled differently, either horizontally (need to store more data) or vertically (need higher throughput).
Read Replicas has a very interesting trait. They do not need persistent storage as they themselves are temporary. Read Replicas within Containers can be bin packed just like any other application by an Orchestrator and backed by a local storage. Write Replicas would not. That would still be resolving only half the problem but would minimize it.