Answers: <p>For the best performance naming many small files in an s3 bucket may not be the best idea unless they user_id is a random string. See how the S3 hashing algorithm works <a href="https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/" rel="nofollow noreferrer">https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/</a></p>
  
  <p>So replacing it by a random hashed value may not be that bad. Of course you need to store the hash in the user profile or you need to calculate it from something. If you add something like profile id or last changed timestamp you can do sha1/md5 of (userid+lastchanged) and if you update timestamp you has the hash of the current avatar. </p>
  
  <p>The older one you can simply delete. CloudFront will remove it from its cache after expiration <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html</a></p>
  
  <p>This will more complex algorithm, but if the costs are the concern, a little bit of extra work seems to be a reasonable approach. Not sure how do you handle the uploads, but it may be a good opportunity to play a bit with step functions which would make sure that the new avatar is uploaded correctly and then the old one is removed as well. </p>
  
  <p>It's kind of a programming question, but I get your point - it is very specific to AWS environment and requires a particular knowledge about how the infrastructure works. </p>
  
Answers: CC BY-SA 4.0
Answers: 2019-11-18T06:46:54.553Z
Answers: Okay, I think you're saying to go with Option 2 because it's not as bad as I thought because if an object goes long enough without being accessed it will be removed from the CloudFront edge nodes? And that storage space in s3 is basically infinite anyway so it's fine to have data in there even if it isn't accessed?
Answers: Chron Bag
Answers: CC BY-SA 4.0
Answers: 2019-11-18T14:57:46.480Z
Answers: yes, option 2. I think I tried to suggest that it should not be that complicated to delete the old avatar at the point where you are uploading the new version, so you do not even waste the space. But it is an optional step.
Answers: petrchpetr
Answers: CC BY-SA 4.0
Answers: 2019-11-18T20:01:32.773Z
Answers: I'm not sure how I would delete the old avatar without it being possible for other malicious users to delete other user's avatars. Considering I send the user the presigned URL for uploading to S3, and then they send back their new avatar name, which I set on the server. (I have to trust that they send back the proper filename). I can't set their avatar in the database when they just request the presigned URL, because the upload might fail, or they might cancel it. So my only option is to set it later, no?
Answers: Chron Bag
Answers: CC BY-SA 4.0
Answers: 2019-11-18T21:05:38.490Z
Answers: This starts to seem like a very unusual design pattern. "I have to trust" for sure sounds like a big design flaw. With the presign url you decide how the object will be named before passing the presign url to the user. You can also send user metadata with that URL (e.g. user-id) https://stackoverflow.com/questions/37614286/aws-s3-add-set-update-specify-user-metadata-with-a-presigned-url then once the object is successfully uploaded it can trigger a lambda. lambda will read the user-id and can update the db, if there is already another avatar in the db it can delete it without any issues.
Answers: petrchpetr
Answers: CC BY-SA 4.0
Answers: 2019-11-18T22:18:35.857Z
Answers: Well, I'm not sure of a way around the trust. You have to update the database column 'avatar' with the name of the object in s3. It would be ideal if I could update the column when generating the presigned URL, but that would be flawed, because the user may just not use the presigned URL. For example, if you update the column and then the user fails their upload to s3, it's too late, because the column is already updated in the database. And you can't just save the presigned URL in the database, because if the user used a different presigned URL, then the filename wouldn't match.
Answers: Chron Bag
Answers: CC BY-SA 4.0
Answers: 2019-11-19T08:12:57.107Z
Answers: Check s3 bucket triggering lambda functions for updating the database name https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html generate the presigned URL for a s3 key (filename) which is a random string, if you need the original name, store it in metadata of that file. That way you don't need to know the name in advance and you update your db only with files which which were correctly updated. Think more asynchronously. Each step should trigger the next one once it finishes - events, SNS, SQS, Step Functions - more cloud fun is comming.
Answers: petrchpetr
Answers: CC BY-SA 4.0
Answers: 2019-11-17T20:47:49.387Z
Answers: 2019-11-17T20:47:49.387Z
Answers: petrchpetr
Body: <p>On my site I have the ability for users to upload avatars (which they upload to s3). They can only have one avatar at the moment, so I currently just have the filename set to their user id.</p>
  
  <p>However, they may replace their avatars, which causes issues. Because the s3 is propagating to CloudFront, I need to invalidate the cache for the filename when they upload their new avatar (because it replaces the old avatar file). But with CloudFront invalidations are expensive.</p>
  
  <p>It almost seems like Amazon would prefer I just not delete the old files, and instead just give the new avatar a new name and leave the old one in there, even though it's no longer being used by anyone. That way I can avoid the invalidation.</p>
  
  <p>The other option is not to use CloudFront at all, and just upload to s3, and make the s3 bucket public. But the problem with this is that CloudFront is much more generous with their GET request costs and amounts (2,000,000 per month with CloudFront and only 20,000 per month with S3 buckets).</p>
  
  <p>Basically, I have these 3 options, and I'm not sure which to pick, given the above:</p>
  
  <p>1) Replace the files in s3 and then invalidate the file in CloudFront</p>
  
  <ul>
  <li><strong>Problem:</strong> invalidations are expensive</li>
  </ul>
  
  <p>2) Just dump more files in the s3 bucket and use CloudFront</p>
  
  <ul>
  <li><strong>Problem:</strong> the old, unused avatars just sit there wasting s3 bucket space</li>
  </ul>
  
  <p>3) Don't use CloudFront, and just replace the file in s3 and then serve it from the bucket</p>
  
  <ul>
  <li><strong>Problem:</strong> CloudFront has cheaper GET requests and better user experience with latency</li>
  </ul>
  
  <p><strong>Edit</strong>: In this case, I'm asking about AWS with s3 and CloudFront. I'm looking to most effectively automate the process by which a user can upload an avatar to my backend service. Specifically, the overwriting of old avatars has me confused. The general process flow of "PreSigned URL -> Upload to s3 -> Propagate via CloudFront" seems an adequate one, but I'm not sure how to accommodate the overwriting of old, no longer used files in this plan, considering the choice seems to be between an expensive invalidation, or the keeping of stale files.</p>
  
Comments: CC BY-SA 4.0
Comments: 2019-11-17T07:54:09.773Z
Comments: Hi there! This question sounds like a vanilla programming problem - can you help us understand how it pertains to DevOps? Are there particular communication, process or automation aspects which you are asking about? Thanks!
Comments: Bruce Becker
Comments: CC BY-SA 4.0
Comments: 2019-11-17T08:10:38.390Z
Comments: I checked with a couple people and they all recommended I post here, since it's the combination of development and operations. Also, there are a lot of other questions about S3 and CloudFront here. Should I post on StackOverflow instead? Or somewhere else?
Comments: Chron Bag
Comments: CC BY-SA 4.0
Comments: 2019-11-17T10:58:50.010Z
Comments: @ChronBag StackOverflow is restricted to programming questions. I agree with you that you post it on DevOps SE. However, please make it more specific as suggested by BruceBecker by addressing his question: `Are there particular communication, process or automation aspects which you are asking about?` by adding a new paragraph to the Q&A.
Comments: 030
Comments: CC BY-SA 4.0
Comments: 2019-11-17T18:47:14.177Z
Comments: Okay, I'm not really sure what that means but I'll try my best.
Comments: Chron Bag
Tags: <amazon-web-services><amazon-s3><amazon-cloudfront><cache>
Title: For storing avatars in s3, which approach has the lowest cost?
UserDisplayName: Chron Bag
ThreadText: # For storing avatars in s3, which approach has the lowest cost?
  
  (Asked by: username_0 on 2019-11-17)
  
  On my site I have the ability for users to upload avatars (which they upload to s3). They can only have one avatar at the moment, so I currently just have the filename set to their user id.
  
  However, they may replace their avatars, which causes issues. Because the s3 is propagating to CloudFront, I need to invalidate the cache for the filename when they upload their new avatar (because it replaces the old avatar file). But with CloudFront invalidations are expensive.
  
  It almost seems like Amazon would prefer I just not delete the old files, and instead just give the new avatar a new name and leave the old one in there, even though it's no longer being used by anyone. That way I can avoid the invalidation.
  
  The other option is not to use CloudFront at all, and just upload to s3, and make the s3 bucket public. But the problem with this is that CloudFront is much more generous with their GET request costs and amounts (2,000,000 per month with CloudFront and only 20,000 per month with S3 buckets).
  
  Basically, I have these 3 options, and I'm not sure which to pick, given the above:
  
  1) Replace the files in s3 and then invalidate the file in CloudFront
  
  * **Problem:** invalidations are expensive
  
  2) Just dump more files in the s3 bucket and use CloudFront
  
  * **Problem:** the old, unused avatars just sit there wasting s3 bucket space
  
  3) Don't use CloudFront, and just replace the file in s3 and then serve it from the bucket
  
  * **Problem:** CloudFront has cheaper GET requests and better user experience with latency
  
  **Edit**: In this case, I'm asking about AWS with s3 and CloudFront. I'm looking to most effectively automate the process by which a user can upload an avatar to my backend service. Specifically, the overwriting of old avatars has me confused. The general process flow of "PreSigned URL -> Upload to s3 -> Propagate via CloudFront" seems an adequate one, but I'm not sure how to accommodate the overwriting of old, no longer used files in this plan, considering the choice seems to be between an expensive invalidation, or the keeping of stale files.
  
  **Tags:** amazon-web-services, amazon-s3, amazon-cloudfront, cache
  
  > **Comments:**
  > **username_1**: Hi there! This question sounds like a vanilla programming problem - can you help us understand how it pertains to DevOps? Are there particular communication, process or automation aspects which you are asking about? Thanks!  
  > **username_0**: I checked with a couple people and they all recommended I post here, since it's the combination of development and operations. Also, there are a lot of other questions about S3 and CloudFront here. Should I post on StackOverflow instead? Or somewhere else?  
  > **username_2**: @username_0 StackOverflow is restricted to programming questions. I agree with you that you post it on DevOps SE. However, please make it more specific as suggested by username_1 by addressing his question: `Are there particular communication, process or automation aspects which you are asking about?` by adding a new paragraph to the Q&A.  
  > **username_0**: Okay, I'm not really sure what that means but I'll try my best.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2019-11-17. Score: 3 | Accepted answer
  
  For the best performance naming many small files in an s3 bucket may not be the best idea unless they user\_id is a random string. See how the S3 hashing algorithm works <https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/>
  
  So replacing it by a random hashed value may not be that bad. Of course you need to store the hash in the user profile or you need to calculate it from something. If you add something like profile id or last changed timestamp you can do sha1/md5 of (userid+lastchanged) and if you update timestamp you has the hash of the current avatar.
  
  The older one you can simply delete. CloudFront will remove it from its cache after expiration <https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html>
  
  This will more complex algorithm, but if the costs are the concern, a little bit of extra work seems to be a reasonable approach. Not sure how do you handle the uploads, but it may be a good opportunity to play a bit with step functions which would make sure that the new avatar is uploaded correctly and then the old one is removed as well.
  
  It's kind of a programming question, but I get your point - it is very specific to AWS environment and requires a particular knowledge about how the infrastructure works.
  
  > **Comments:**
  > **username_0**: Okay, I think you're saying to go with Option 2 because it's not as bad as I thought because if an object goes long enough without being accessed it will be removed from the CloudFront edge nodes? And that storage space in s3 is basically infinite anyway so it's fine to have data in there even if it isn't accessed?  
  > **username_3**: yes, option 2. I think I tried to suggest that it should not be that complicated to delete the old avatar at the point where you are uploading the new version, so you do not even waste the space. But it is an optional step.  
  > **username_0**: I'm not sure how I would delete the old avatar without it being possible for other malicious users to delete other user's avatars. Considering I send the user the presigned URL for uploading to S3, and then they send back their new avatar name, which I set on the server. (I have to trust that they send back the proper filename). I can't set their avatar in the database when they just request the presigned URL, because the upload might fail, or they might cancel it. So my only option is to set it later, no?  
  > **username_3**: This starts to seem like a very unusual design pattern. "I have to trust" for sure sounds like a big design flaw. With the presign url you decide how the object will be named before passing the presign url to the user. You can also send user metadata with that URL (e.g. user-id) https://stackoverflow.com/questions/37614286/aws-s3-add-set-update-specify-user-metadata-with-a-presigned-url then once the object is successfully uploaded it can trigger a lambda. lambda will read the user-id and can update the db, if there is already another avatar in the db it can delete it without any issues.  
  > **username_0**: Well, I'm not sure of a way around the trust. You have to update the database column 'avatar' with the name of the object in s3. It would be ideal if I could update the column when generating the presigned URL, but that would be flawed, because the user may just not use the presigned URL. For example, if you update the column and then the user fails their upload to s3, it's too late, because the column is already updated in the database. And you can't just save the presigned URL in the database, because if the user used a different presigned URL, then the filename wouldn't match.  
  > **username_3**: Check s3 bucket triggering lambda functions for updating the database name https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html generate the presigned URL for a s3 key (filename) which is a random string, if you need the original name, store it in metadata of that file. That way you don't need to know the name in advance and you update your db only with files which which were correctly updated. Think more asynchronously. Each step should trigger the next one once it finishes - events, SNS, SQS, Step Functions - more cloud fun is comming.  
  
  ---
  
  