Answers.Body: <p>Further inspecting the source code of the downloaded page, I found the following:</p>
  
  <pre class="lang-html prettyprint-override"><code>&lt;li id="nav-sequences"
      class="module-load transcript"
      wname="sequences"
      href="/rest/widget/transcript/K06C4.12/sequences"&gt;
      &lt;span class="ui-icon ui-icon-close module-close" wname="sequences"
      &lt;/span&gt;
      Sequences
  &lt;/li&gt;
  </code></pre>
  
  <p>This suggested me to have a look at <a href="http://www.wormbase.org/rest/widget/transcript/K06C4.12/sequences" rel="nofollow noreferrer">http://www.wormbase.org/rest/widget/transcript/K06C4.12/sequences</a> and I obtained a page containing the elements found in the DOM inspector.</p>
  
  <p>This can be parsed, for instance using the <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/" rel="nofollow noreferrer"><code>bs4</code> python module</a>:</p>
  
  <pre class="lang-python prettyprint-override"><code># To get the page
  import requests
  # To parse the page content
  from bs4 import BeautifulSoup
  # To get this into Biopython
  from io import StringIO
  from Bio import SeqIO
  
  rest_template = "http://www.wormbase.org/rest/widget/transcript/{accession}/sequences"
  rest = rest_template.format(accession="K06C4.12")
  # This did not work with "html.parser"
  soup = BeautifulSoup(requests.get(rest).text, "html5lib")
  
  def generate_fasta(soup):
      name = None
      for div in soup.find_all("div"):
          if "ui-helper-hidden" in div["class"]:
              if div["id"] == "content":
                  fasta = SeqIO.read(StringIO(div.text), "fasta")
                  description_fields = fasta.description.split()
                  if len(description_fields) == 4:
                      fasta.id = "_".join(description_fields[:2])
                      if name is None:
                          name = fasta.name
                  elif len(description_fields) == 2:
                      # This assumes that the "conceptual_translation"
                      # is after the "unspliced + UTR" or "spliced + UTR"
                      # Otherwise, one would need to pass
                      # the accession as function parameter
                      fasta.id = f"{name}_translation"
                  else:
                      fasta.id = "_".join(description_fields)
                  yield (fasta.id, fasta)
  
  records_dict = dict(generate_fasta(soup))
  print(records_dict['K06C4.12_spliced'])
  print(records_dict['K06C4.12_translation'])
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>ID: K06C4.12_spliced
  Name: K06C4.12
  Description: K06C4.12 spliced + UTR
  Number of features: 0
  Seq('ATGGCCCCACCAAAGCCGTCTGCTAAAGGAGCAAAGAAAGCCGCCAAGACCGTC...TAA', SingleLetterAlphabet())
  ID: K06C4.12_translation
  Name: conceptual
  Description: conceptual translation
  Number of features: 0
  Seq('MAPPKPSAKGAKKAAKTVTKPKDGKKRRHARKESYSVYIYRVLKQVHPDTGVSS...SK*', SingleLetterAlphabet())
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-24T12:35:52.727Z
Answers.LastActivityDate: 2017-11-24T14:12:06.163Z
Answers.LastEditDate: 2017-11-24T14:12:06.163Z
Answers.UserDisplayName: bli
Body: <p>I'm working on the <em>C. elegans</em> model organism, for which there exists an online biological information database: Wormbase.</p>
  
  <p>I'm able to manually search for a transcript (say "K06C4.12") and this gets me the following URL:
  <a href="http://www.wormbase.org/species/c_elegans/cds/K06C4.12" rel="nofollow noreferrer">http://www.wormbase.org/species/c_elegans/cds/K06C4.12</a></p>
  
  <p>Say I have this transcript ID: "F08G2.2". I guess that I can form a valid similar URL, as follows:
  <a href="http://www.wormbase.org/species/c_elegans/cds/F08G2.2" rel="nofollow noreferrer">http://www.wormbase.org/species/c_elegans/cds/F08G2.2</a></p>
  
  <p>Indeed, it works as expected.</p>
  
  <p>Now, I would like to go further and be able to download in an automated way what can be obtained by clicking on one of the "download" buttons in the "Sequence" section of the above pages. More specifically, I'm interested in the "spliced + UTR" and "conceptual translation" fasta sequences. </p>
  
  <p>If I use the DOM inspector in Firefox (Shift-Ctrl-C), I can locate things like the following:</p>
  
  <pre class="lang-html prettyprint-override"><code>&lt;div class="button ui-corner-all generate-file-download sequence-download"&gt;
      &lt;span class="ui-icon ui-icon-arrowthickstop-1-s"&gt;&lt;/span&gt;
      &lt;span&gt;download&lt;/span&gt;
      &lt;div id="content" class="ui-helper-hidden"&gt;
          &gt;K06C4.12 spliced + UTR
          ATGGCCCCACCAAAGCCGTCTGCTAAAGGAGCAAAGAAAGCCGCCAAGACCGTCACCAAGCCAAAGGACGGAAAGAAGAG
          ACGTCATGCCCGTAAAGAATCCTACTCCGTGTACATCTACCGTGTGCTCAAGCAGGTTCATCCAGACACTGGAGTCTCTT
          CCAAGGCTATGTCCATCATGAACTCTTTCGTCAACGACGTCTTCGAGCGTATCGCTGCTGAGGCATCTCGTCTTGCTCAC
          TACAACAAGAGATCCACCATCTCTTCTCGCGAAATTCAAACTGCCGTCCGTTTGATTCTTCCAGGAGAGCTTGCCAAGCA
          CGCCGTCTCTGAGGGAACCAAGGCTGTCACCAAGTACACCTCTAGCAAGTAA
      &lt;/div&gt;
      &lt;div id="filename" class="ui-helper-hidden"&gt;K06C4.12conceptual translation.fasta&lt;/div&gt;
      &lt;div id="content" class="ui-helper-hidden"&gt;
          &gt;conceptual translation
          MAPPKPSAKGAKKAAKTVTKPKDGKKRRHARKESYSVYIYRVLKQVHPDTGVSSKAMSIMNSFVNDVFERIAAEASRLAH
          YNKRSTISSREIQTAVRLILPGELAKHAVSEGTKAVTKYTSSK*
      &lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="toggle "&gt;
      &lt;span style="float:left" class="ui-icon ui-icon-triangle-1-s"&gt;&lt;/span&gt;
      view conceptual translation (123 aa)
  &lt;/div&gt;
  &lt;div class="returned sequence-container" style="display:block"&gt;
      &gt;conceptual translation
      &lt;br&gt;
      MAPPKPSAKGAKKAAKTVTKPKDGKKRRHARKESYSVYIYRVLKQVHPDTGVSSKAMSIMNSFVNDVFERIAAEASRLAH
      &lt;br&gt;
      YNKRSTISSREIQTAVRLILPGELAKHAVSEGTKAVTKYTSSK*
  &lt;/div&gt;
  </code></pre>
  
  <p>I don't find these if I simply download the page with <code>wget</code> and search in the source code, so they must be dynamically generated.</p>
  
  <p><strong>How could I proceed to get the sequences?</strong></p>
  
  <p>The <a href="http://www.wormbase.org/about/userguide/for_developers/API-REST" rel="nofollow noreferrer">documentation about Wormbases's REST API</a> is not very detailed, so I don't understand how to form an URL that would give me the desired sequences.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-24T13:28:38.907Z
Comments.Text: Did you try to download the whole database and search there? (I am not sure if you need large amounts of data or not ...)
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-24T13:32:09.980Z
Comments.Text: I don't know where to download it, neither how it is structured. I don't need a large amount of data, so I hope querying the site should be OK.
Comments.UserDisplayName: bli
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-24T13:33:41.650Z
Comments.Text: To download it they said it is [here](http://www.wormbase.org/about/userguide#3--10). There should be a ftp site, so you could try that too.
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-24T13:45:57.317Z
Comments.Text: Indeed, they even provide a scary shell script (first time I see a "goto" instruction in a shell script) to install the database: ftp://ftp.wormbase.org/pub/wormbase/releases/current-production-release/acedb/INSTALL
  I'm not sure I'm ready to delve into this for the moment...
Comments.UserDisplayName: bli
Tags: <python><public-databases><data-retrieval><api>
Title: Automating sequence download from Wormbase
ThreadText: # Automating sequence download from Wormbase
  
  (Asked by: username_0 on 2017-11-24)
  
  I'm working on the *C. elegans* model organism, for which there exists an online biological information database: Wormbase.
  
  I'm able to manually search for a transcript (say "K06C4.12") and this gets me the following URL:
  <http://www.wormbase.org/species/c_elegans/cds/K06C4.12>
  
  Say I have this transcript ID: "F08G2.2". I guess that I can form a valid similar URL, as follows:
  <http://www.wormbase.org/species/c_elegans/cds/F08G2.2>
  
  Indeed, it works as expected.
  
  Now, I would like to go further and be able to download in an automated way what can be obtained by clicking on one of the "download" buttons in the "Sequence" section of the above pages. More specifically, I'm interested in the "spliced + UTR" and "conceptual translation" fasta sequences.
  
  If I use the DOM inspector in Firefox (Shift-Ctrl-C), I can locate things like the following:
  
  ```html
  <div class="button ui-corner-all generate-file-download sequence-download">
      <span class="ui-icon ui-icon-arrowthickstop-1-s"></span>
      <span>download</span>
      <div id="content" class="ui-helper-hidden">
          >K06C4.12 spliced + UTR
          ATGGCCCCACCAAAGCCGTCTGCTAAAGGAGCAAAGAAAGCCGCCAAGACCGTCACCAAGCCAAAGGACGGAAAGAAGAG
          ACGTCATGCCCGTAAAGAATCCTACTCCGTGTACATCTACCGTGTGCTCAAGCAGGTTCATCCAGACACTGGAGTCTCTT
          CCAAGGCTATGTCCATCATGAACTCTTTCGTCAACGACGTCTTCGAGCGTATCGCTGCTGAGGCATCTCGTCTTGCTCAC
          TACAACAAGAGATCCACCATCTCTTCTCGCGAAATTCAAACTGCCGTCCGTTTGATTCTTCCAGGAGAGCTTGCCAAGCA
          CGCCGTCTCTGAGGGAACCAAGGCTGTCACCAAGTACACCTCTAGCAAGTAA
      </div>
      <div id="filename" class="ui-helper-hidden">K06C4.12conceptual translation.fasta</div>
      <div id="content" class="ui-helper-hidden">
          >conceptual translation
          MAPPKPSAKGAKKAAKTVTKPKDGKKRRHARKESYSVYIYRVLKQVHPDTGVSSKAMSIMNSFVNDVFERIAAEASRLAH
          YNKRSTISSREIQTAVRLILPGELAKHAVSEGTKAVTKYTSSK*
      </div>
  </div>
  <div class="toggle ">
      <span style="float:left" class="ui-icon ui-icon-triangle-1-s"></span>
      view conceptual translation (123 aa)
  </div>
  <div class="returned sequence-container" style="display:block">
      >conceptual translation
      <br>
      MAPPKPSAKGAKKAAKTVTKPKDGKKRRHARKESYSVYIYRVLKQVHPDTGVSSKAMSIMNSFVNDVFERIAAEASRLAH
      <br>
      YNKRSTISSREIQTAVRLILPGELAKHAVSEGTKAVTKYTSSK*
  </div>
  
  ```
  
  I don't find these if I simply download the page with `wget` and search in the source code, so they must be dynamically generated.
  
  **How could I proceed to get the sequences?**
  
  The [documentation about Wormbases's REST API](http://www.wormbase.org/about/userguide/for_developers/API-REST) is not very detailed, so I don't understand how to form an URL that would give me the desired sequences.
  
  **Tags:** python, public-databases, data-retrieval, api
  
  > **Comments:**
  > **username_1**: Did you try to download the whole database and search there? (I am not sure if you need large amounts of data or not ...)  
  > **username_0**: I don't know where to download it, neither how it is structured. I don't need a large amount of data, so I hope querying the site should be OK.  
  > **username_1**: To download it they said it is [here](http://www.wormbase.org/about/userguide#3--10). There should be a ftp site, so you could try that too.  
  > **username_0**: Indeed, they even provide a scary shell script (first time I see a "goto" instruction in a shell script) to install the database: ftp://ftp.wormbase.org/pub/wormbase/releases/current-production-release/acedb/INSTALL I'm not sure I'm ready to delve into this for the moment...  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2017-11-24. Score: 4
  
  Further inspecting the source code of the downloaded page, I found the following:
  
  ```html
  <li id="nav-sequences"
      class="module-load transcript"
      wname="sequences"
      href="/rest/widget/transcript/K06C4.12/sequences">
      <span class="ui-icon ui-icon-close module-close" wname="sequences"
      </span>
      Sequences
  </li>
  
  ```
  
  This suggested me to have a look at <http://www.wormbase.org/rest/widget/transcript/K06C4.12/sequences> and I obtained a page containing the elements found in the DOM inspector.
  
  This can be parsed, for instance using the [`bs4` python module](https://www.crummy.com/software/BeautifulSoup/bs4/doc/):
  
  ```python
  # To get the page
  import requests
  # To parse the page content
  from bs4 import BeautifulSoup
  # To get this into Biopython
  from io import StringIO
  from Bio import SeqIO
  
  rest_template = "http://www.wormbase.org/rest/widget/transcript/{accession}/sequences"
  rest = rest_template.format(accession="K06C4.12")
  # This did not work with "html.parser"
  soup = BeautifulSoup(requests.get(rest).text, "html5lib")
  
  def generate_fasta(soup):
      name = None
      for div in soup.find_all("div"):
          if "ui-helper-hidden" in div["class"]:
              if div["id"] == "content":
                  fasta = SeqIO.read(StringIO(div.text), "fasta")
                  description_fields = fasta.description.split()
                  if len(description_fields) == 4:
                      fasta.id = "_".join(description_fields[:2])
                      if name is None:
                          name = fasta.name
                  elif len(description_fields) == 2:
                      # This assumes that the "conceptual_translation"
                      # is after the "unspliced + UTR" or "spliced + UTR"
                      # Otherwise, one would need to pass
                      # the accession as function parameter
                      fasta.id = f"{name}_translation"
                  else:
                      fasta.id = "_".join(description_fields)
                  yield (fasta.id, fasta)
  
  records_dict = dict(generate_fasta(soup))
  print(records_dict['K06C4.12_spliced'])
  print(records_dict['K06C4.12_translation'])
  
  ```
  
  Output:
  
  ```
  ID: K06C4.12_spliced
  Name: K06C4.12
  Description: K06C4.12 spliced + UTR
  Number of features: 0
  Seq('ATGGCCCCACCAAAGCCGTCTGCTAAAGGAGCAAAGAAAGCCGCCAAGACCGTC...TAA', SingleLetterAlphabet())
  ID: K06C4.12_translation
  Name: conceptual
  Description: conceptual translation
  Number of features: 0
  Seq('MAPPKPSAKGAKKAAKTVTKPKDGKKRRHARKESYSVYIYRVLKQVHPDTGVSS...SK*', SingleLetterAlphabet())
  
  ```
  
  
  ---
  
  