Answers.Body: <p>Apparently there is no <strong>100% reliable</strong> solution, but in the most cases, you should be able to use the <a href="https://github.com/enasequence/enaBrowserTools" rel="nofollow noreferrer">enaBrowserTools</a> python scripts (enaDataGet) to quite reliably download data.</p>
  
  <p>The run ERR2135453 is a special case because the sample submission (ERS1939590) for it has been cancelled. The solution for these reads is to download the data manually from the ftp location.</p>
  
  <p>This answer is based on kind assistance of Suran from ebi helpdesk.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-11T15:27:03.167Z
Answers.Text: Did you reach them through the email?
Answers.UserDisplayName: llrs
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-11T17:15:28.647Z
Answers.Text: Do, they are usually very responsive and helpful.
Answers.UserDisplayName: Konrad Rudolph
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-11T14:38:27.583Z
Answers.LastActivityDate: 2018-05-25T11:52:34.537Z
Answers.LastEditDate: 2018-05-25T11:52:34.537Z
Answers.UserDisplayName: Kamil S Jaron
Answers.Body: <p>Here are two python functions I use for getting stuff from ENA based on an accession:
  </p>
  
  <pre class="lang-py prettyprint-override"><code>import os
  import itertools
  from urllib.request import urlopen
  
  def fetch_ENA_files(accession):
      """Get the names of the files matching the ENA accession"""
  
      # Get the paths to the files
      url = "http://www.ebi.ac.uk/ena/data/warehouse/filereport?accession=%s&amp;result=read_run&amp;fields=fastq_ftp" \
            % accession
  
      try:
          paths = urlopen(url).readlines()[1:]
      except:
          E.debug("couldn't access %s" % url)
          raise
  
      paths = list(itertools.chain.from_iterable(
          [p.strip().split(";") for p in paths]))
      filenames = [os.path.basename(x) for x in paths]
      dl_paths = [os.path.join(*x.split("/")[1:]) for x in paths]
      return filenames, dl_paths
  
  def fetch_ENA(dl_path, outdir, protocol="ascp"):
      """Fetch fastq from ENA given accession"""
  
      if protocol == "ascp":
          statement = """ascp -QT -P33001 -l 100M -i $ASCP_KEY_PATH
                         era-fasp@fasp.sra.ebi.ac.uk:/%(dl_path)s %(outdir)s """ % locals()
      elif protocol == "http":
          fn = os.path.basename(dl_path)
          outfile = os.path.join(outdir, fn)
          statement = "wget -O %(outfile)s ftp://ftp.sra.ebi.ac.uk:/%(dl_path)s" % locals()
  
      return statement
  </code></pre>
  
  <p><code>fetch_ENA_files</code> queries the ENA database for the names of all the files associated with a accession and returns a tuple of lists. The first entry in the tuple contains the filenames (which you might need later in your application), the second contains a list of paths on the ENA servers where those files can be found. </p>
  
  <p><code>fetch_ENA(dl_path, outdir, protocol)</code> takes a download path and downloads it to <code>outdir</code> using with ftp or ascp.</p>
  
  <p>You might use it like:</p>
  
  <pre class="lang-py prettyprint-override"><code>filenames, paths = fetch_ENA_files("ERR2135453")
  for dlp in paths:
      bash_statement = fetch_ENA(dlp, "downloads", "ascp")
      os.system(bash_statement)
  </code></pre>
  
  <p>(we don't we actaully use our modules to send it to the cluster)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-11T17:06:32.320Z
Answers.Text: I had to decode the paths from bytes to strings to get this to work. It doesn't actually work for the accession in question so I guess that's further evidence that there's no way to reliably fetch data sets that are too recent to be in SRA
Answers.UserDisplayName: heathobrien
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-14T08:47:22.537Z
Answers.Text: Sorry, should have pointed out that this is python 2 not 3, hence the encoding problems.
Answers.UserDisplayName: Ian Sudbery
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-11T15:44:23.840Z
Answers.LastActivityDate: 2018-05-11T17:06:35.770Z
Answers.LastEditDate: 2018-05-11T17:06:35.770Z
Answers.UserDisplayName: Ian Sudbery
Body: <p>I am trying to build a workflow that gets data automatically from databases of sequencing reads (SRA - sequence read archive, ENA - european nucleotide archive). Till now I was pulling everything from SRA and my script have worked, but now I encountered a first entry that is only in ENA (<a href="https://www.ebi.ac.uk/ena/data/view/ERR2135453" rel="nofollow noreferrer">ERR2135453</a>), <a href="https://www.ncbi.nlm.nih.gov/sra/?term=ERR2135453" rel="nofollow noreferrer">not in SRA</a>. And I found that my download script has its limitations.</p>
  
  <p>My bash script looks like this (requires aspera <code>ascp</code> and <code>fastq-dump</code>) and I wanted to make changes that are commented now</p>
  
  <pre><code>#!/bin/bash
  
  ACCESION=$1
  OPENSSH=/my/part/to/key/asperaweb_id_dsa.openssh
  
  # if [ ${ACCESION:0:3} == SRA ]; then
      URL=anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/
  # else
  #    URL=ena_url
  # fi    
  
  ascp -QT -i $OPENSSH $URL
  
  fastq-dump data/"$ACCESION".sra --outdir data/ --split-files --gzip
  rm data/"$ACCESION".sra
  </code></pre>
  
  <p>The problem is in getting ENA url. Unlike SRA url that I can easily get from the accession (<code>anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/</code>), the ftp server of ENA is bit more confusing. The url corresponding to ERR2135453 accession is <code>ftp://ftp.sra.ebi.ac.uk//vol1/err/ERR213/003/ERR2135453</code>. Where <code>003</code> there comes from? Also not all reads are in <code>err</code> subdirectory. </p>
  
  <p>Is there an easy way how to download reads from ENA knowing just the accession number?</p>
  
  <p>--- edit ---</p>
  
  <p>I found on the relevant ENA <a href="https://www.ebi.ac.uk/ena/browse/read-download#downloading_files_aspera" rel="nofollow noreferrer">site about downloading data using aspera</a> that ftp urls should be possible to derive only from accession also in ENA.</p>
  
  <blockquote>
    <p>Submitted read data files are organised by submission accession number
    under vol1/ directory in ftp.sra.ebi.ac.uk:</p>
    
    <p><a href="ftp://ftp.sra.ebi.ac.uk/vol1/" rel="nofollow noreferrer">ftp://ftp.sra.ebi.ac.uk/vol1/</a>/</p>
    
    <p>where  contains the first 6 letters and
    numbers of the SRA Submission accession.</p>
    
    <p>For example, the files submitted in the SRA Submission ERA007448 are
    available at: <a href="ftp://ftp.sra.ebi.ac.uk/vol1/ERA007/ERA007448/" rel="nofollow noreferrer">ftp://ftp.sra.ebi.ac.uk/vol1/ERA007/ERA007448/</a>.</p>
  </blockquote>
  
  <p>The reason why there is this strange <code>003</code> will probably be that it's quite recent submission and it has not been properly categorized yet (other accessions I tried were on "expected" urls).</p>
  
  <p>However, the problem still persist. Is there a way how to fetch data of new submissions using accessions only?</p>
  
Tags: <data-download><sratoolkit>
Title: A reliable fetcher of short read using SRA/ENA accession
ThreadText: # A reliable fetcher of short read using SRA/ENA accession
  
  (Asked by: username_0 on 2018-05-10)
  
  I am trying to build a workflow that gets data automatically from databases of sequencing reads (SRA - sequence read archive, ENA - european nucleotide archive). Till now I was pulling everything from SRA and my script have worked, but now I encountered a first entry that is only in ENA ([ERR2135453](https://www.ebi.ac.uk/ena/data/view/ERR2135453)), [not in SRA](https://www.ncbi.nlm.nih.gov/sra/?term=ERR2135453). And I found that my download script has its limitations.
  
  My bash script looks like this (requires aspera `ascp` and `fastq-dump`) and I wanted to make changes that are commented now
  
  ```
  #!/bin/bash
  
  ACCESION=$1
  OPENSSH=/my/part/to/key/asperaweb_id_dsa.openssh
  
  # if [ ${ACCESION:0:3} == SRA ]; then
      URL=anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/
  # else
  #    URL=ena_url
  # fi    
  
  ascp -QT -i $OPENSSH $URL
  
  fastq-dump data/"$ACCESION".sra --outdir data/ --split-files --gzip
  rm data/"$ACCESION".sra
  
  ```
  
  The problem is in getting ENA url. Unlike SRA url that I can easily get from the accession (`anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/`), the ftp server of ENA is bit more confusing. The url corresponding to ERR2135453 accession is `ftp://ftp.sra.ebi.ac.uk//vol1/err/ERR213/003/ERR2135453`. Where `003` there comes from? Also not all reads are in `err` subdirectory.
  
  Is there an easy way how to download reads from ENA knowing just the accession number?
  
  --- edit ---
  
  I found on the relevant ENA [site about downloading data using aspera](https://www.ebi.ac.uk/ena/browse/read-download#downloading_files_aspera) that ftp urls should be possible to derive only from accession also in ENA.
  
  > Submitted read data files are organised by submission accession number
  > under vol1/ directory in ftp.sra.ebi.ac.uk:
  >
  > <ftp://ftp.sra.ebi.ac.uk/vol1/>/
  >
  > where contains the first 6 letters and
  > numbers of the SRA Submission accession.
  >
  > For example, the files submitted in the SRA Submission ERA007448 are
  > available at: <ftp://ftp.sra.ebi.ac.uk/vol1/ERA007/ERA007448/>.
  
  The reason why there is this strange `003` will probably be that it's quite recent submission and it has not been properly categorized yet (other accessions I tried were on "expected" urls).
  
  However, the problem still persist. Is there a way how to fetch data of new submissions using accessions only?
  
  **Tags:** data-download, sratoolkit
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2018-05-11. Score: 1 | Accepted answer
  
  Apparently there is no **100% reliable** solution, but in the most cases, you should be able to use the [enaBrowserTools](https://github.com/enasequence/enaBrowserTools) python scripts (enaDataGet) to quite reliably download data.
  
  The run ERR2135453 is a special case because the sample submission (ERS1939590) for it has been cancelled. The solution for these reads is to download the data manually from the ftp location.
  
  This answer is based on kind assistance of Suran from ebi helpdesk.
  
  > **Comments:**
  > **username_1**: Did you reach them through the email?  
  > **username_2**: Do, they are usually very responsive and helpful.  
  
  ---
  
  ### Answer by username_3 on 2018-05-11. Score: 1
  
  Here are two python functions I use for getting stuff from ENA based on an accession:
  
  ```py
  import os
  import itertools
  from urllib.request import urlopen
  
  def fetch_ENA_files(accession):
      """Get the names of the files matching the ENA accession"""
  
      # Get the paths to the files
      url = "http://www.ebi.ac.uk/ena/data/warehouse/filereport?accession=%s&result=read_run&fields=fastq_ftp" \
            % accession
  
      try:
          paths = urlopen(url).readlines()[1:]
      except:
          E.debug("couldn't access %s" % url)
          raise
  
      paths = list(itertools.chain.from_iterable(
          [p.strip().split(";") for p in paths]))
      filenames = [os.path.basename(x) for x in paths]
      dl_paths = [os.path.join(*x.split("/")[1:]) for x in paths]
      return filenames, dl_paths
  
  def fetch_ENA(dl_path, outdir, protocol="ascp"):
      """Fetch fastq from ENA given accession"""
  
      if protocol == "ascp":
          statement = """ascp -QT -P33001 -l 100M -i $ASCP_KEY_PATH
                         era-fasp@fasp.sra.ebi.ac.uk:/%(dl_path)s %(outdir)s """ % locals()
      elif protocol == "http":
          fn = os.path.basename(dl_path)
          outfile = os.path.join(outdir, fn)
          statement = "wget -O %(outfile)s ftp://ftp.sra.ebi.ac.uk:/%(dl_path)s" % locals()
  
      return statement
  
  ```
  
  `fetch_ENA_files` queries the ENA database for the names of all the files associated with a accession and returns a tuple of lists. The first entry in the tuple contains the filenames (which you might need later in your application), the second contains a list of paths on the ENA servers where those files can be found.
  
  `fetch_ENA(dl_path, outdir, protocol)` takes a download path and downloads it to `outdir` using with ftp or ascp.
  
  You might use it like:
  
  ```py
  filenames, paths = fetch_ENA_files("ERR2135453")
  for dlp in paths:
      bash_statement = fetch_ENA(dlp, "downloads", "ascp")
      os.system(bash_statement)
  
  ```
  
  (we don't we actaully use our modules to send it to the cluster)
  
  > **Comments:**
  > **username_4**: I had to decode the paths from bytes to strings to get this to work. It doesn't actually work for the accession in question so I guess that's further evidence that there's no way to reliably fetch data sets that are too recent to be in SRA  
  > **username_3**: Sorry, should have pointed out that this is python 2 not 3, hence the encoding problems.  
  
  ---
  
  