Answers.Body: <p>If I understand correctly, your question boils down to: parsing a bunch of numbers in a row from a .csv file gives a different result from parsing a bunch of numbers as <em>rows</em> of a text file.  I that right?  I suggest that you look at the source of your text file.  For example, was the text file created on a Windows host (running bash, or in the command terminal?)?</p>
  
  <p>Try using this modified version of the unix cat command:
  <code>cat -et text_file.txt</code></p>
  
  <p>The very end of each line will have a '\$' sign and that will show you if there are any extra non-visible characters in your rows.  For example, a DOS/Windows formatted file will usually have a ^M at the end of each row before the '$'.  If you don't strip off those line endings then they get added to your queries, even if you cannot see them.  You can strip off those linefeed (or are they carriage returns?) using a Perl one-liner like:
  <code>perl -pe 's/\r//' badfile.txt &gt; goodfile.txt</code></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-02T00:46:21.807Z
Answers.Text: Thanks, I suspect that between generating these files on my school's servers through bash to downloading them locally and manipulating them in Excel, that there must've been '$' characters like you said.  Since then, I've learned to do what I was doing in Excel on the command line and think I'll be able to avoid this altogether.
Answers.UserDisplayName: ljs
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-27T03:54:22.750Z
Answers.LastActivityDate: 2018-07-27T03:54:22.750Z
Answers.UserDisplayName: mdperry
Body: <p>I've been searching TaxIDs against NCBI's Taxonomy DB to get taxonomic lineages for species.  I have successfully done this for 1,000's of TaxIDs that were returned to me by Blast+ blasts in a CSV.  (The first column in the CSV was the TaxID for the top hit of a sequence, and the other columns were for GI, Scientific Name, e-value, etc.).  I simply wrote a script that would take the first field, which is always the TaxID, and search it against Taxonomy to get lineages for the species those TaxIDs represented.  Everything worked fine!</p>
  
  <p>However, some of the records returned from my original blast (that gave me the TaxIDs in the first place) would return two or more TaxIDs per record.  (I suspect because a sequence matched equally well with two or more different species?)  That's fine, but I had to remove these few exceptions before I searched the rest of the regular results against Taxonomy.  So I've got most of my data/lineages, but for about 1:10,0000 records that gave multiple TaxIDs, I would like to know the lineages of the multiple species that the blast suggested.</p>
  
  <p>I now took all of those "more-than-one-taxid" hits, separated them out into columns in an excel sheet and saved it as a CSV, and am trying to search these against Taxonomy with essentially the same script I used for the rest of the data, but am getting an error for <em>every</em> taxID I search (see below).</p>
  
  <p><strong>Complete Error</strong> (for each taxid searched against Taxonomy):</p>
  
  <pre><code>400 Bad Request
  &amp;rettype=full&amp;retmode=xml&amp;edirect_os=linux&amp;edirect=7.90&amp;tool=edirect&amp;email=x@xxx.localdomain'my&amp;id=464126
  Result of do_post http request is
  $VAR1 = bless( {
                   '_protocol' =&gt; 'HTTP/1.1',
                   '_content' =&gt; '&lt;?xml version="1.0" encoding="UTF-8" ?&gt;
  &lt;!DOCTYPE eEfetchResult PUBLIC "-//NLM//DTD efetch 20131226//EN" "https://eutils.ncbi.nlm.nih.gov/eutils/dtd/20131226/efetch.dtd"&gt;
  &lt;eFetchResult&gt;
   &lt;ERROR&gt;WebEnv parameter is required&lt;/ERROR&gt;
  &lt;/eFetchResult&gt;
  ',
                   '_rc' =&gt; 400,
                   '_headers' =&gt; bless( {
                                          'connection' =&gt; 'close',
                                          'set-cookie' =&gt; 'ncbi_sid=9FF573148F458940_CEC6SID; domain=.nih.gov; path=/; expires=Thu, 14 Mar 2019 22:36:03 GMT',
                                          'cache-control' =&gt; 'private',
                                          'access-control-allow-origin' =&gt; '*',
                                          'date' =&gt; 'Wed, 14 Mar 2018 22:36:03 GMT',
                                          'vary' =&gt; 'Accept-Encoding',
                                          'client-ssl-cert-issuer' =&gt; '/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA',
                                          'client-ssl-cipher' =&gt; 'ECDHE-RSA-AES256-GCM-SHA384',
                                          'client-peer' =&gt; '130.14.29.110:443',
                                          'strict-transport-security' =&gt; 'max-age=31536000; includeSubDomains; preload',
                                          'client-date' =&gt; 'Wed, 14 Mar 2018 22:36:03 GMT',
                                          'content-type' =&gt; 'text/xml; charset=UTF-8',
                                          'x-xss-protection' =&gt; '1; mode=block',
                                          'client-transfer-encoding' =&gt; [
                                                                          'chunked'
                                                                        ],
                                          'server' =&gt; 'Finatra',
                                          'client-ssl-socket-class' =&gt; 'IO::Socket::SSL',
                                          'client-response-num' =&gt; 1,
                                          'ncbi-phid' =&gt; 'DAF6EB0134DF5AEC039F11D248910BE7.1',
                                          'ncbi-sid' =&gt; '9FF573148F458940_CEC6SID',
                                          '::std_case' =&gt; {
                                                            'set-cookie' =&gt; 'Set-Cookie',
                                                            'access-control-allow-origin' =&gt; 'Access-Control-Allow-Origin',
                                                            'client-ssl-cert-issuer' =&gt; 'Client-SSL-Cert-Issuer',
                                                            'client-ssl-cipher' =&gt; 'Client-SSL-Cipher',
                                                            'client-peer' =&gt; 'Client-Peer',
                                                            'strict-transport-security' =&gt; 'Strict-Transport-Security',
                                                            'client-date' =&gt; 'Client-Date',
                                                            'x-xss-protection' =&gt; 'X-XSS-Protection',
                                                            'client-transfer-encoding' =&gt; 'Client-Transfer-Encoding',
                                                            'client-ssl-socket-class' =&gt; 'Client-SSL-Socket-Class',
                                                            'client-response-num' =&gt; 'Client-Response-Num',
                                                            'ncbi-phid' =&gt; 'NCBI-PHID',
                                                            'ncbi-sid' =&gt; 'NCBI-SID',
                                                            'x-ua-compatible' =&gt; 'X-UA-Compatible',
                                                            'client-ssl-cert-subject' =&gt; 'Client-SSL-Cert-Subject',
                                                            'content-security-policy' =&gt; 'Content-Security-Policy'
                                                          },
                                          'x-ua-compatible' =&gt; 'IE=Edge',
                                          'client-ssl-cert-subject' =&gt; '/C=US/ST=Maryland/L=Bethesda/O=National Library of Medicine/CN=*.ncbi.nlm.nih.gov',
                                          'content-security-policy' =&gt; 'upgrade-insecure-requests'
                                        }, 'HTTP::Headers' ),
                   '_msg' =&gt; 'Bad Request',
                   '_request' =&gt; bless( {
  &amp;rettype=full&amp;retmode=xml&amp;edirect_os=linux&amp;edirect=7.90&amp;tool=edirect&amp;email=x@xxx.localdomain',
                                          '_uri' =&gt; bless( do{\(my $o = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi')}, 'URI::https' ),
                                          '_headers' =&gt; bless( {
                                                                 'user-agent' =&gt; 'libwww-perl/6.05',
                                                                 'content-type' =&gt; 'application/x-www-form-urlencoded',
                                                                 '::std_case' =&gt; {
                                                                                   'if-ssl-cert-subject' =&gt; 'If-SSL-Cert-Subject'
                                                                                 }
                                                               }, 'HTTP::Headers' ),
                                          '_method' =&gt; 'POST',
                                          '_uri_canonical' =&gt; $VAR1-&gt;{'_request'}{'_uri'}
                                        }, 'HTTP::Request' )
                 }, 'HTTP::Response' );
  </code></pre>
  
  <p><strong>What I've done:</strong></p>
  
  <p>Here is a sample of the data I used that worked correctly and yielded lineages when searching the taxIDs (taxIDs in 1st column):</p>
  
  <pre><code>549298,Contig1,Francisella halioticida,5.44e-126,gi|1213460577|gb|CP022132.1|,CP022132
  240395,Contig2,Glandirana emeljanovi,4.74e-131,gi|1025812536|gb|KU641020.1|,KU641020
  1069680,Contig3,Pneumocystis murina B123,2.34e-85,gi|1124120202|ref|XM_019757798.1|,XM_019757798
  3871,Contig4,Lupinus angustifolius,1.89e-93,gi|1239725835|gb|CP023132.1|,CP023132
  </code></pre>
  
  <p>Here is a sample of the <strong>data that I am trying to get to work</strong> (when I search these manually against Taxonomy from their web page they seem to work and give lineages; it's just that I have just enough of them to make a non-programmatic solution really time consuming and tedious):</p>
  
  <pre><code>258234
  1316279
  1316315
  710996
  711000
  225338
  985895
  710996
  1382731
  77133
  344338
  </code></pre>
  
  <p>This is the business end of <strong>the script that I used that worked for the  working data</strong>:</p>
  
  <pre><code>#!/bin/bash
  while IFS=',' read -r f1 f2 f3 f4 f5 f6
  do      
          #Firstly echo the Contig# (Query Seq. ID) followed by lineage information.
          echo -n "$f2," &gt;&gt; $2
  
          #Now get the lineage, pipe to xtract and parse XML for readability in output file
          efetch -db taxonomy \
          -id $f1 \
          -format xml \
          | xtract -pattern Taxon \
                  -tab ',' -sep ',' \
                  -def "NONE" \
                  -element TaxId,ScientificName \
                  -division LineageEx \
                  -group Taxon \
                  -if Rank -equals superkingdom \
                  -or Rank -equals kingdom \
                  ...
                  -or Rank -equals species \
                  -tab ',' -sep ',' \
                  -def "NOT_AVAILABLE" \
                  -element Rank,ScientificName \
                  &gt;&gt; $2
  done &lt; "$1"
  </code></pre>
  
  <p>This is the slightly altered script that I am trying to use to make the not-working set of data yield results (and that produces the error):</p>
  
  <pre><code>#!/bin/bash
  while IFS='' read -r line || [[ -n "$line" ]]; do
          efetch -db taxonomy \
          -id $line \
          -format xml \
          | xtract -pattern Taxon \
                  -tab ',' -sep ',' \
                  -def "NONE" \
                  -element TaxId,ScientificName \
                  -division LineageEx \
                  -group Taxon \
                  -if Rank -equals superkingdom \
                  -or Rank -equals kingdom \
                  ...
                  -or Rank -equals species \
                  -or Rank -equals subspecies \
                  -tab ',' -sep ',' \
                  -def "NOT_AVAILABLE" \
                  -element Rank,ScientificName \
                  &gt;&gt; $2
  done &lt; "$1"
  </code></pre>
  
  <p>As you can see, <strong>the only difference</strong> is that I am reading in entire lines for the TaxIDs list because it is simply a list with line breaks between the IDs as opposed to a true CSV with other values as in the example that works.  I've saved the taxIDs list as both a CSV and TXT and it doesn't make a difference.  Also: I can echo what $line is in the second script and it is exactly I'd expect--just the numbers of the taxID, <strong>so why would I be getting a 400 error for bad syntax on my query if it looks exactly like what I am sending in the first, working script?</strong></p>
  
  <p>Thanks!</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-15T14:13:01.480Z
Comments.Text: Hello, nice question. Just, could you please tame ([edit]) a bit the error log? It's very hard to look look at.
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-30T20:51:18.827Z
Comments.Text: Aside from your specific question slightly, you might be interested in looking at the `NCBITaxa()` tool in the ETE3 package. It basically does what your script does, but provides lots of other useful methods for querying the ncbi taxa database.
Comments.UserDisplayName: Joe Healey
Tags: <taxonomy><shell>
Title: Using a Bash Script to search TaxIDs against NCBI's Taxonomy yields "400 Bad Request" error?
ThreadText: # Using a Bash Script to search TaxIDs against NCBI's Taxonomy yields "400 Bad Request" error?
  
  (Asked by: username_0 on 2018-03-14)
  
  I've been searching TaxIDs against NCBI's Taxonomy DB to get taxonomic lineages for species. I have successfully done this for 1,000's of TaxIDs that were returned to me by Blast+ blasts in a CSV. (The first column in the CSV was the TaxID for the top hit of a sequence, and the other columns were for GI, Scientific Name, e-value, etc.). I simply wrote a script that would take the first field, which is always the TaxID, and search it against Taxonomy to get lineages for the species those TaxIDs represented. Everything worked fine!
  
  However, some of the records returned from my original blast (that gave me the TaxIDs in the first place) would return two or more TaxIDs per record. (I suspect because a sequence matched equally well with two or more different species?) That's fine, but I had to remove these few exceptions before I searched the rest of the regular results against Taxonomy. So I've got most of my data/lineages, but for about 1:10,0000 records that gave multiple TaxIDs, I would like to know the lineages of the multiple species that the blast suggested.
  
  I now took all of those "more-than-one-taxid" hits, separated them out into columns in an excel sheet and saved it as a CSV, and am trying to search these against Taxonomy with essentially the same script I used for the rest of the data, but am getting an error for *every* taxID I search (see below).
  
  **Complete Error** (for each taxid searched against Taxonomy):
  
  ```
  400 Bad Request
  &rettype=full&retmode=xml&edirect_os=linux&edirect=7.90&tool=edirect&email=x@xxx.localdomain'my&id=464126
  Result of do_post http request is
  $VAR1 = bless( {
                   '_protocol' => 'HTTP/1.1',
                   '_content' => '<?xml version="1.0" encoding="UTF-8" ?>
  <!DOCTYPE eEfetchResult PUBLIC "-//NLM//DTD efetch 20131226//EN" "https://eutils.ncbi.nlm.nih.gov/eutils/dtd/20131226/efetch.dtd">
  <eFetchResult>
   <ERROR>WebEnv parameter is required</ERROR>
  </eFetchResult>
  ',
                   '_rc' => 400,
                   '_headers' => bless( {
                                          'connection' => 'close',
                                          'set-cookie' => 'ncbi_sid=9FF573148F458940_CEC6SID; domain=.nih.gov; path=/; expires=Thu, 14 Mar 2019 22:36:03 GMT',
                                          'cache-control' => 'private',
                                          'access-control-allow-origin' => '*',
                                          'date' => 'Wed, 14 Mar 2018 22:36:03 GMT',
                                          'vary' => 'Accept-Encoding',
                                          'client-ssl-cert-issuer' => '/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA',
                                          'client-ssl-cipher' => 'ECDHE-RSA-AES256-GCM-SHA384',
                                          'client-peer' => '130.14.29.110:443',
                                          'strict-transport-security' => 'max-age=31536000; includeSubDomains; preload',
                                          'client-date' => 'Wed, 14 Mar 2018 22:36:03 GMT',
                                          'content-type' => 'text/xml; charset=UTF-8',
                                          'x-xss-protection' => '1; mode=block',
                                          'client-transfer-encoding' => [
                                                                          'chunked'
                                                                        ],
                                          'server' => 'Finatra',
                                          'client-ssl-socket-class' => 'IO::Socket::SSL',
                                          'client-response-num' => 1,
                                          'ncbi-phid' => 'DAF6EB0134DF5AEC039F11D248910BE7.1',
                                          'ncbi-sid' => '9FF573148F458940_CEC6SID',
                                          '::std_case' => {
                                                            'set-cookie' => 'Set-Cookie',
                                                            'access-control-allow-origin' => 'Access-Control-Allow-Origin',
                                                            'client-ssl-cert-issuer' => 'Client-SSL-Cert-Issuer',
                                                            'client-ssl-cipher' => 'Client-SSL-Cipher',
                                                            'client-peer' => 'Client-Peer',
                                                            'strict-transport-security' => 'Strict-Transport-Security',
                                                            'client-date' => 'Client-Date',
                                                            'x-xss-protection' => 'X-XSS-Protection',
                                                            'client-transfer-encoding' => 'Client-Transfer-Encoding',
                                                            'client-ssl-socket-class' => 'Client-SSL-Socket-Class',
                                                            'client-response-num' => 'Client-Response-Num',
                                                            'ncbi-phid' => 'NCBI-PHID',
                                                            'ncbi-sid' => 'NCBI-SID',
                                                            'x-ua-compatible' => 'X-UA-Compatible',
                                                            'client-ssl-cert-subject' => 'Client-SSL-Cert-Subject',
                                                            'content-security-policy' => 'Content-Security-Policy'
                                                          },
                                          'x-ua-compatible' => 'IE=Edge',
                                          'client-ssl-cert-subject' => '/C=US/ST=Maryland/L=Bethesda/O=National Library of Medicine/CN=*.ncbi.nlm.nih.gov',
                                          'content-security-policy' => 'upgrade-insecure-requests'
                                        }, 'HTTP::Headers' ),
                   '_msg' => 'Bad Request',
                   '_request' => bless( {
  &rettype=full&retmode=xml&edirect_os=linux&edirect=7.90&tool=edirect&email=x@xxx.localdomain',
                                          '_uri' => bless( do{\(my $o = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi')}, 'URI::https' ),
                                          '_headers' => bless( {
                                                                 'user-agent' => 'libwww-perl/6.05',
                                                                 'content-type' => 'application/x-www-form-urlencoded',
                                                                 '::std_case' => {
                                                                                   'if-ssl-cert-subject' => 'If-SSL-Cert-Subject'
                                                                                 }
                                                               }, 'HTTP::Headers' ),
                                          '_method' => 'POST',
                                          '_uri_canonical' => $VAR1->{'_request'}{'_uri'}
                                        }, 'HTTP::Request' )
                 }, 'HTTP::Response' );
  
  ```
  
  **What I've done:**
  
  Here is a sample of the data I used that worked correctly and yielded lineages when searching the taxIDs (taxIDs in 1st column):
  
  ```
  549298,Contig1,Francisella halioticida,5.44e-126,gi|1213460577|gb|CP022132.1|,CP022132
  240395,Contig2,Glandirana emeljanovi,4.74e-131,gi|1025812536|gb|KU641020.1|,KU641020
  1069680,Contig3,Pneumocystis murina B123,2.34e-85,gi|1124120202|ref|XM_019757798.1|,XM_019757798
  3871,Contig4,Lupinus angustifolius,1.89e-93,gi|1239725835|gb|CP023132.1|,CP023132
  
  ```
  
  Here is a sample of the **data that I am trying to get to work** (when I search these manually against Taxonomy from their web page they seem to work and give lineages; it's just that I have just enough of them to make a non-programmatic solution really time consuming and tedious):
  
  ```
  258234
  1316279
  1316315
  710996
  711000
  225338
  985895
  710996
  1382731
  77133
  344338
  
  ```
  
  This is the business end of **the script that I used that worked for the working data**:
  
  ```
  #!/bin/bash
  while IFS=',' read -r f1 f2 f3 f4 f5 f6
  do      
          #Firstly echo the Contig# (Query Seq. ID) followed by lineage information.
          echo -n "$f2," >> $2
  
          #Now get the lineage, pipe to xtract and parse XML for readability in output file
          efetch -db taxonomy \
          -id $f1 \
          -format xml \
          | xtract -pattern Taxon \
                  -tab ',' -sep ',' \
                  -def "NONE" \
                  -element TaxId,ScientificName \
                  -division LineageEx \
                  -group Taxon \
                  -if Rank -equals superkingdom \
                  -or Rank -equals kingdom \
                  ...
                  -or Rank -equals species \
                  -tab ',' -sep ',' \
                  -def "NOT_AVAILABLE" \
                  -element Rank,ScientificName \
                  >> $2
  done < "$1"
  
  ```
  
  This is the slightly altered script that I am trying to use to make the not-working set of data yield results (and that produces the error):
  
  ```
  #!/bin/bash
  while IFS='' read -r line || [[ -n "$line" ]]; do
          efetch -db taxonomy \
          -id $line \
          -format xml \
          | xtract -pattern Taxon \
                  -tab ',' -sep ',' \
                  -def "NONE" \
                  -element TaxId,ScientificName \
                  -division LineageEx \
                  -group Taxon \
                  -if Rank -equals superkingdom \
                  -or Rank -equals kingdom \
                  ...
                  -or Rank -equals species \
                  -or Rank -equals subspecies \
                  -tab ',' -sep ',' \
                  -def "NOT_AVAILABLE" \
                  -element Rank,ScientificName \
                  >> $2
  done < "$1"
  
  ```
  
  As you can see, **the only difference** is that I am reading in entire lines for the TaxIDs list because it is simply a list with line breaks between the IDs as opposed to a true CSV with other values as in the example that works. I've saved the taxIDs list as both a CSV and TXT and it doesn't make a difference. Also: I can echo what $line is in the second script and it is exactly I'd expect--just the numbers of the taxID, **so why would I be getting a 400 error for bad syntax on my query if it looks exactly like what I am sending in the first, working script?**
  
  Thanks!
  
  **Tags:** taxonomy, shell
  
  > **Comments:**
  > **username_1**: Hello, nice question. Just, could you please tame ([edit]) a bit the error log? It's very hard to look look at.  
  > **username_2**: Aside from your specific question slightly, you might be interested in looking at the `NCBITaxa()` tool in the ETE3 package. It basically does what your script does, but provides lots of other useful methods for querying the ncbi taxa database.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2018-07-27. Score: 3 | Accepted answer
  
  If I understand correctly, your question boils down to: parsing a bunch of numbers in a row from a .csv file gives a different result from parsing a bunch of numbers as *rows* of a text file. I that right? I suggest that you look at the source of your text file. For example, was the text file created on a Windows host (running bash, or in the command terminal?)?
  
  Try using this modified version of the unix cat command:
  `cat -et text_file.txt`
  
  The very end of each line will have a '\$' sign and that will show you if there are any extra non-visible characters in your rows. For example, a DOS/Windows formatted file will usually have a ^M at the end of each row before the '$'. If you don't strip off those line endings then they get added to your queries, even if you cannot see them. You can strip off those linefeed (or are they carriage returns?) using a Perl one-liner like:
  `perl -pe 's/\r//' badfile.txt > goodfile.txt`
  
  > **Comments:**
  > **username_0**: Thanks, I suspect that between generating these files on my school's servers through bash to downloading them locally and manipulating them in Excel, that there must've been '$' characters like you said.  Since then, I've learned to do what I was doing in Excel on the command line and think I'll be able to avoid this altogether.  
  
  ---
  
  