mounting an s3 bucket in ec2 and using transparently as a mnt point
I'd suggest using a separately-mounted EBS volume. I tried doing the same thing for some movie files. Access to S3 was slow, and S3 has some limitations like not being able to rename files, no real directory structure, etc.
You can set up EBS volumes in a RAID5 configuration and add space as you need it.
I have a webapp (call it myapp.com) that allows users to upload files.  The webapp will be deployed on Amazon EC2 instance.  I would like to serve these files back out to the webapp consumers via an s3 bucket based domain (i.e. uploads.myapp.com).  
When the user uploads the files, I can easily drop them in into a folder called "site_uploads" on the local ec2 instance.  However, since my ec2 instance has finite storage, with a lot of uploads, the ec2 file system will fill up quickly.  
It would be great if the ec2 instance could mount and s3 bucket as the "site_upload" directory.  So that uploads to the EC2 "site_upload" directory automatically end up on uploads.myapp.com (and my webapp can use template tags to make sure the links for this uploaded content is based on that s3 backed domain).  This also gives me scalable file serving, as request for files hits s3 and not my ec2 instance.  Also, it makes it easy for my webapp to perform scaling/resizing of the images that appear locally in "site_upload" but are actually on s3.
I'm looking at s3fs, but judging from the comments, it doesn't look like a fully baked solution.  I'm looking for a non-commercial solution.
FYI, The webapp is written in django, not that that changes the particulars too much.