Answers.Body: <p>If you just want the number of unique headers, you can do this:</p>
  
  <pre><code>grep '&gt;' reference.fasta | sort | uniq | wc -l
  </code></pre>
  
  <p>If you want a list of the unique headers, you can do this:</p>
  
  <pre><code>grep '&gt;' reference.fasta | sort | uniq
  </code></pre>
  
  <p>If you want a histogram of how many times each header occurs, you can do this:</p>
  
  <pre><code>grep '&gt;' reference.fasta | sort | uniq -c | awk '{printf("%s\t%s\n", <span class="math-container">$1, $</span>2)}'
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-10T03:14:54.157Z
Answers.LastActivityDate: 2018-10-10T08:24:08.857Z
Answers.LastEditDate: 2018-10-10T08:24:08.857Z
Answers.UserDisplayName: conchoecia
Answers.Body: <p>The <code>uniq</code> command expects sorted input. Interestingly, the <code>sort</code> command actually has a "unique" option, <code>-u</code>, which means <code>uniq</code> is not strictly needed. For the fastest processing, you can look for the '>' character at the start of lines with grep:</p>
  
  <pre><code>grep '^&gt;' reference.fasta | sort -u &gt; reference_headers_unique.fasta
  </code></pre>
  
  <p>For returning the <em>number</em> of unique lines, pipe through <code>wc -l</code>:</p>
  
  <pre><code>grep '^&gt;' reference.fasta | sort -u | wc -l
  </code></pre>
  
  <p>For more information about regular expressions, see <a href="https://www.gnu.org/software/findutils/manual/html_node/find_html/grep-regular-expression-syntax.html#grep-regular-expression-syntax" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-10T09:39:45.163Z
Answers.Text: This seems like the best solution given, just note it assumes that the desired output order for the unique headers is the order provided by `sort` (which may not be stable depending on the locale)
Answers.UserDisplayName: Chris_Rands
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-10T04:19:22.000Z
Answers.LastActivityDate: 2018-10-10T04:19:22.000Z
Answers.UserDisplayName: gringer
Answers.Body: <p>If, for some reason, you need to keep the original order of the input file, so you only want the <em>first</em> occurrence of each header, you can do:</p>
  
  <pre><code>awk '/^&gt;/ &amp;&amp; !a[$0]++' reference.fasta &gt; headers
  </code></pre>
  
  <p>Or, to get the number only:</p>
  
  <pre><code>awk '/^&gt;/ &amp;&amp; !a[$0]++{k++}; END{print k}' reference.fasta 
  </code></pre>
  
  <p>However, these only make sense if you need to keep the order of the original file. If not, use <a href="https://bioinformatics.stackexchange.com/a/5188/298">Gringer's <code>grep</code> approach</a> which will be much faster (<code>reference.fasta</code> is hg19):</p>
  
  <pre><code><span class="math-container">$ time awk '/^&gt;/ &amp;&amp; !a[$</span>0]++{k++}; END{print k}' reference.fasta 
  93
  
  real    0m34.510s
  user    0m31.861s
  sys     0m2.634s
  
  $ time grep '^&gt;' reference.fasta | sort -u | wc -l
  93
  
  real    0m16.597s
  user    0m0.820s
  sys     0m3.725s
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-12T07:53:07.073Z
Answers.Text: +1 for order capturing approach and benchmark, I guess ripgrep might be even faster
Answers.UserDisplayName: Chris_Rands
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-10T12:57:22.167Z
Answers.LastActivityDate: 2018-10-11T07:58:07.463Z
Answers.LastEditDate: 2018-10-11T07:58:07.463Z
Answers.UserDisplayName: terdon
Body: <p>I tried to use the following command</p>
  
  <pre><code>uniq -u reference.fasta &gt;&gt; reference_uniq.fasta
  </code></pre>
  
  <p>I'd like a count of the unique headers.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-10T08:39:15.590Z
Comments.Text: does it matter what the output order of the unique headers is?
Comments.UserDisplayName: Chris_Rands
Tags: <fasta><linux>
Title: finding unique headers in a fasta file using linux command line
ThreadText: # finding unique headers in a fasta file using linux command line
  
  (Asked by: username_0 on 2018-10-10)
  
  I tried to use the following command
  
  ```
  uniq -u reference.fasta >> reference_uniq.fasta
  
  ```
  
  I'd like a count of the unique headers.
  
  **Tags:** fasta, linux
  
  > **Comments:**
  > **username_1**: does it matter what the output order of the unique headers is?  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2018-10-10. Score: 5
  
  If you just want the number of unique headers, you can do this:
  
  ```
  grep '>' reference.fasta | sort | uniq | wc -l
  
  ```
  
  If you want a list of the unique headers, you can do this:
  
  ```
  grep '>' reference.fasta | sort | uniq
  
  ```
  
  If you want a histogram of how many times each header occurs, you can do this:
  
  ```
  grep '>' reference.fasta | sort | uniq -c | awk '{printf("%s\t%s\n", $1, $2)}'
  
  ```
  
  
  ---
  
  ### Answer by username_3 on 2018-10-10. Score: 3
  
  The `uniq` command expects sorted input. Interestingly, the `sort` command actually has a "unique" option, `-u`, which means `uniq` is not strictly needed. For the fastest processing, you can look for the '>' character at the start of lines with grep:
  
  ```
  grep '^>' reference.fasta | sort -u > reference_headers_unique.fasta
  
  ```
  
  For returning the *number* of unique lines, pipe through `wc -l`:
  
  ```
  grep '^>' reference.fasta | sort -u | wc -l
  
  ```
  
  For more information about regular expressions, see [here](https://www.gnu.org/software/findutils/manual/html_node/find_html/grep-regular-expression-syntax.html#grep-regular-expression-syntax).
  
  > **Comments:**
  > **username_1**: This seems like the best solution given, just note it assumes that the desired output order for the unique headers is the order provided by `sort` (which may not be stable depending on the locale)  
  
  ---
  
  ### Answer by username_4 on 2018-10-10. Score: 1
  
  If, for some reason, you need to keep the original order of the input file, so you only want the *first* occurrence of each header, you can do:
  
  ```
  awk '/^>/ && !a[$0]++' reference.fasta > headers
  
  ```
  
  Or, to get the number only:
  
  ```
  awk '/^>/ && !a[$0]++{k++}; END{print k}' reference.fasta 
  
  ```
  
  However, these only make sense if you need to keep the order of the original file. If not, use [username_3's `grep` approach](https://bioinformatics.stackexchange.com/a/5188/298) which will be much faster (`reference.fasta` is hg19):
  
  ```
  $ time awk '/^>/ && !a[$0]++{k++}; END{print k}' reference.fasta 
  93
  
  real    0m34.510s
  user    0m31.861s
  sys     0m2.634s
  
  $ time grep '^>' reference.fasta | sort -u | wc -l
  93
  
  real    0m16.597s
  user    0m0.820s
  sys     0m3.725s
  
  ```
  
  > **Comments:**
  > **username_1**: +1 for order capturing approach and benchmark, I guess ripgrep might be even faster  
  
  ---
  
  