Answers.Body: <p>You could set up a <a href="https://aws.amazon.com/premiumsupport/knowledge-center/read-access-objects-s3-bucket/" rel="nofollow noreferrer">public read-only</a> bucket in an S3 account and put your files there. URLs would look like:</p>
  
  <pre><code>https://<span class="math-container">${username}.s3.amazonaws.com/public/$</span>{filename}
  </code></pre>
  
  <p>For samtools, the request for a BAM file's index should be built off that URL. For IGV, I think you can specify separate, custom URLs for the data and the index resources.</p>
  
  <p>Public buckets are available to all, without need for credentials. So that might be an issue to think about, with respect to <a href="https://aws.amazon.com/s3/pricing/" rel="nofollow noreferrer">billing for web requests and data transfer</a>.</p>
  
  <p>I'm still not 100% certain that samtools/tabix support https requests reliably (I have a prior question on here about that), but that might be one approach.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-18T10:21:10.137Z
Answers.LastActivityDate: 2020-04-24T21:59:39.770Z
Answers.LastEditDate: 2020-04-24T21:59:39.770Z
Answers.UserDisplayName: Alex Reynolds
Answers.Body: <p>Doo you have your heart set on AWS, or are you open to using Google Cloud Platform (GCP)? </p>
  
  <p>If you're open to using GCP, there are some advantages. For example GATK can read and write directly to and from Google storage buckets, and IGV can use your login credentials to access data in Google storage as described here: </p>
  
  <p><a href="https://googlegenomics.readthedocs.io/en/latest/use_cases/browse_genomic_data/igv.html" rel="nofollow noreferrer">https://googlegenomics.readthedocs.io/en/latest/use_cases/browse_genomic_data/igv.html</a></p>
  
  <p>Note that if you make the data public, by default you're going to get billed if people download it. Not sure how that's handled on AWS, but on GCP you can set the bucket (or just the file) to be "requester-pays", ie the person who downloads is the person who pays for it: </p>
  
  <p><a href="https://cloud.google.com/storage/docs/requester-pays" rel="nofollow noreferrer">https://cloud.google.com/storage/docs/requester-pays</a></p>
  
  <p>In general there are several ways to interact with Google buckets. </p>
  
  <ul>
  <li><p>web-based file browser (example is a public bucket I manage): <a href="https://console.cloud.google.com/storage/browser/genomics-in-the-cloud" rel="nofollow noreferrer">https://console.cloud.google.com/storage/browser/genomics-in-the-cloud</a></p></li>
  <li><p>command-line access through the cloud sdk toolkit <code>gsutil</code>: <a href="https://cloud.google.com/storage/docs/gsutil" rel="nofollow noreferrer">https://cloud.google.com/storage/docs/gsutil</a></p></li>
  </ul>
  
  <p><em>Disclosure: I work at the Broad Institute, which uses a lot of GCP infrastructure; I do not get any compensation from Google Cloud, but I recently wrote a book about Genomics in the Cloud that uses GCP.</em> </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-24T18:20:07.027Z
Answers.LastActivityDate: 2020-04-24T18:20:07.027Z
Answers.UserDisplayName: Geraldine_VdAuwera
Answers.Body: <p>I have recently open sourced a Java NIO SPI for S3 that allows direct reads from S3 using Java applications that use the NIO model. This allows applications like GATK and some parts of Picard to understand <code>s3://</code> URIs.</p>
  <p>It is used as a drop in so as long as the jar is on your classpath or in your JRE <code>ext</code> folder then Java will delegate nio operations to this lib when it &quot;sees&quot; an <code>s3://</code> URI. It will use the credentials and configuration of your current profile so no additional config is require and no recompilation of an application is needed.</p>
  <p>The github repo is <a href="https://github.com/awslabs/aws-java-nio-spi-for-s3" rel="nofollow noreferrer">https://github.com/awslabs/aws-java-nio-spi-for-s3</a></p>
  <p><em>Disclosure: I am the author of the library, employed by the AWS Health AI Genomics Team, and have financial interests in Amazon</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-10T16:56:02.450Z
Answers.LastActivityDate: 2022-04-10T16:56:02.450Z
Answers.UserDisplayName: Mark
Body: <p>What's the best/easiest way for accessing .bam/.cram files from S3?</p>
  
  <p>I have .bam/.bai .cram/.crai and .bed + .gff files that I want to make available easily to others so they can browse in IGV and other solutions from their computers. There seem to be different solutions after cursory Googling, what would you recommend?</p>
  
  <p>Here is what I've found so far:</p>
  
  <p>IGV:</p>
  
  <ul>
  <li><a href="https://twitter.com/igvteam/status/1187766079457644544" rel="nofollow noreferrer">https://twitter.com/igvteam/status/1187766079457644544</a></li>
  <li><a href="https://umccr.org/blog/igv-amazon-backend-setup/" rel="nofollow noreferrer">https://umccr.org/blog/igv-amazon-backend-setup/</a></li>
  </ul>
  
  <p>Samtools: samtools directly supports AWS</p>
  
  <ul>
  <li><a href="http://www.htslib.org/doc/htslib-s3-plugin.html" rel="nofollow noreferrer">http://www.htslib.org/doc/htslib-s3-plugin.html</a></li>
  </ul>
  
  <pre><code>The full URL format is:
  
      s3[+SCHEME]://[ID[:SECRET[:TOKEN]]@]BUCKET/PATH
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-16T08:12:06.873Z
Comments.Text: samtools directly supports AWS these days: http://www.htslib.org/doc/htslib-s3-plugin.html
Comments.UserDisplayName: Devon Ryan
Tags: <sam><samtools><igv><cram><aws>
Title: Accessing .bam/.cram files from AWS S3?
ThreadText: # Accessing .bam/.cram files from AWS S3?
  
  (Asked by: username_0 on 2020-04-16)
  
  What's the best/easiest way for accessing .bam/.cram files from S3?
  
  I have .bam/.bai .cram/.crai and .bed + .gff files that I want to make available easily to others so they can browse in IGV and other solutions from their computers. There seem to be different solutions after cursory Googling, what would you recommend?
  
  Here is what I've found so far:
  
  IGV:
  
  * <https://twitter.com/igvteam/status/1187766079457644544>
  * <https://umccr.org/blog/igv-amazon-backend-setup/>
  
  Samtools: samtools directly supports AWS
  
  * <http://www.htslib.org/doc/htslib-s3-plugin.html>
  
  ```
  The full URL format is:
  
      s3[+SCHEME]://[ID[:SECRET[:TOKEN]]@]BUCKET/PATH
  
  ```
  
  **Tags:** sam, samtools, igv, cram, aws
  
  > **Comments:**
  > **username_1**: samtools directly supports AWS these days: http://www.htslib.org/doc/htslib-s3-plugin.html  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2020-04-18. Score: 3
  
  You could set up a [public read-only](https://aws.amazon.com/premiumsupport/knowledge-center/read-access-objects-s3-bucket/) bucket in an S3 account and put your files there. URLs would look like:
  
  ```
  https://${username}.s3.amazonaws.com/public/${filename}
  
  ```
  
  For samtools, the request for a BAM file's index should be built off that URL. For IGV, I think you can specify separate, custom URLs for the data and the index resources.
  
  Public buckets are available to all, without need for credentials. So that might be an issue to think about, with respect to [billing for web requests and data transfer](https://aws.amazon.com/s3/pricing/).
  
  I'm still not 100% certain that samtools/tabix support https requests reliably (I have a prior question on here about that), but that might be one approach.
  
  
  ---
  
  ### Answer by username_3 on 2020-04-24. Score: 1
  
  Doo you have your heart set on AWS, or are you open to using Google Cloud Platform (GCP)?
  
  If you're open to using GCP, there are some advantages. For example GATK can read and write directly to and from Google storage buckets, and IGV can use your login credentials to access data in Google storage as described here:
  
  <https://googlegenomics.readthedocs.io/en/latest/use_cases/browse_genomic_data/igv.html>
  
  Note that if you make the data public, by default you're going to get billed if people download it. Not sure how that's handled on AWS, but on GCP you can set the bucket (or just the file) to be "requester-pays", ie the person who downloads is the person who pays for it:
  
  <https://cloud.google.com/storage/docs/requester-pays>
  
  In general there are several ways to interact with Google buckets.
  
  * web-based file browser (example is a public bucket I manage): <https://console.cloud.google.com/storage/browser/genomics-in-the-cloud>
  * command-line access through the cloud sdk toolkit `gsutil`: <https://cloud.google.com/storage/docs/gsutil>
  
  *Disclosure: I work at the Broad Institute, which uses a lot of GCP infrastructure; I do not get any compensation from Google Cloud, but I recently wrote a book about Genomics in the Cloud that uses GCP.*
  
  
  ---
  
  ### Answer by username_4 on 2022-04-10. Score: 1
  
  I have recently open sourced a Java NIO SPI for S3 that allows direct reads from S3 using Java applications that use the NIO model. This allows applications like GATK and some parts of Picard to understand `s3://` URIs.
  
  It is used as a drop in so as long as the jar is on your classpath or in your JRE `ext` folder then Java will delegate nio operations to this lib when it "sees" an `s3://` URI. It will use the credentials and configuration of your current profile so no additional config is require and no recompilation of an application is needed.
  
  The github repo is <https://github.com/awslabs/aws-java-nio-spi-for-s3>
  
  *Disclosure: I am the author of the library, employed by the AWS Health AI Genomics Team, and have financial interests in Amazon*
  
  
  ---
  
  