In praise of S3
I work in a bioinformatics lab that develops data visualizations for genetics. A typical file for a whole human genome is in the 50-100GB range. These files are heavily indexed and streaming-friendly, but still rather unwieldy to work with.<p>Our most common flow involves files stored on S3. The app will download the index, then use sampling and streaming techniques to pull from the full file. This works well because S3 supports range requests.<p>Where S3 falls down for us is sharing. I&#x27;m not aware of any easy way to share S3 files with email. You either have to create a signed URL (which has a max expiration), make the file public, or use IAM, which I&#x27;m pretty sure requires all users to have AWS accounts. That&#x27;s a non-starter.<p>Google Drive is the opposite. It&#x27;s better for sharing. Just drop in an email address, or share a permanent link. But it no longer supports range requests, so it&#x27;s essentially walled off from the rest of the web.<p>What&#x27;s crazy to me is that neither of these services provide a non-authenticated way to use a CLI tool to download&#x2F;sync a directory tree, even if it&#x27;s public. rclone is probably the best tool in this space, and it requires you to configure &quot;remotes&quot; ahead of time. You can&#x27;t just point it at a public bucket and have it download.<p>I think there&#x27;s still room for improvement and competition in the cloud storage space.