One use case would be one that came up a while back here on HN -- The person had a large binary blob that they needed to load up every time their Lambda function ran. If they could mount EFS from Lambda, they could put the large binary blob there.
Depending on how they wrote their code, if the Lambda could seek to the right part of the file then this would work, but if it has to load the entire file into RAM anyway, then it wouldn't gain them much except perhaps a slightly easier way to get the file into RAM.
Couldn't you just download the portion of the one file you want from S3? Unless EFS is significantly faster...
From a caching perspective, you wouldn't seem to gain much in lamda. You can cache in memory/tmpfs until your container eventually dies, and then you likely aren't on the same machine so there's no EFS level caching to take advantage of.
Right that was my point. You most likely have to load the whole thing into memory, but in the rare case that you can seek directly to the right spot in the file, ECS might have an advantage for you in Lambda.
Yeah, I just meant that S3 has partial downloads, so you can still seek.
EFS/NFS makes a lot of sense I think when you have a lot of random/unpredictable access across large numbers of files. You could do the same thing with FUSE/S3 but across lots of smaller files/accesses I would think the overhead adds up faster.
You mention HPC and Big Data processing as use cases for EFS; Lambda is also very well suited for big processing tasks you want to execute quickly with a large amount of concurrency. You often need persistence across executions, for example when breaking tasks into multi-function workflows, and EFS would seem to simplify this over manually using S3 for it.
I would like to have a media pipeline where a media goes through QC check, thumbnail, preview generation and some other processing which are independent of each other. My media can range from 1-15GB, so if I can store it in EFS which can be mounted in Lambda, I can use various Lambda functions which can simply read from a common storage without downloading every time from S3.
I want to run the Elm compiler in a Lambda, and maintain persistent environments so that subsequent builds can benefit from keeping the libraries and intermediate files available.
