Hope everybody is safe and happy.
I have given a AWS task as follow.
Requirement: As soon as file gets uploaded in s3 bucket,it should be zipped and must be uploaded back to another s3 bucket.
I am able to complete this task using lambda(python) but it involves lots of disk IO. So ,i am looking for a solution where i will not require to store incoming s3 object at /tmp/ folder. As we all know , lambda provides very less memory and storage(500MB). Hence, wants to avoid this approach.
So, does anybody aware of how to zip the incoming s3 file on the fly? I am only aware that, it possible to do with the help of streaming of s3 object. But not able to find how it can be achieved end to end using python language.
Thanks in advance.
You can use python generators in the Lambda to zip the s3 files and put the zip file back in another s3 as the generators doesn't occupy memory.
How to create zip file and upload back to s3 using lambda python?