Answers.Body: <p>You can use <code>awk</code> here, by setting the input (<code>RS</code>) and output (<code>ORS</code>) record separator, this is what defines a &quot;line&quot; in awk, to <code>&gt;</code>. Then, tell it to print records that do not match <code>Sequence unavailable</code>. For instance, using this input file:</p>
  <pre><code>$ cat file
  &gt;foo
  AGAGTAGATAGAAGATA
  &gt;bar
  AGAGATAGAGAGATATA
  &gt;baz
  Sequence unavailable
  &gt;bar1
  AGAGATAGAGAGATATA
  &gt;asld
  Sequence unavailable
  &gt;asd
  ACTTC
  </code></pre>
  <p>You can do:</p>
  <pre><code>$ awk  'BEGIN{RS=ORS=&quot;&gt;&quot;}!/Sequence unavailable/' file 
  &gt;foo
  AGAGTAGATAGAAGATA
  &gt;bar
  AGAGATAGAGAGATATA
  &gt;bar1
  AGAGATAGAGAGATATA
  &gt;asd
  ACTTC
  &gt;
  </code></pre>
  <p>That will add a trailing <code>&gt;</code>, so you need one more step to remove it:</p>
  <pre><code>$ awk  'BEGIN{RS=ORS=&quot;&gt;&quot;}!/Sequence unavailable/' file | sed 's/&gt;$//'
  &gt;foo
  AGAGTAGATAGAAGATA
  &gt;bar
  AGAGATAGAGAGATATA
  &gt;bar1
  AGAGATAGAGAGATATA
  &gt;asd
  ACTTC
  </code></pre>
  <hr />
  <p>Alternatively, using the <code>FastaToTbl</code> and <code>TblToFasta</code> scripts I have posted <a href="https://bioinformatics.stackexchange.com/a/2651/298">here</a>, you can do:</p>
  <pre><code>$ FastaToTbl file | grep -v 'Sequence unavailable' | TblToFasta 
  &gt;foo 
  AGAGTAGATAGAAGATA
  &gt;bar 
  AGAGATAGAGAGATATA
  &gt;bar1 
  AGAGATAGAGAGATATA
  &gt;asd 
  ACTTC
  </code></pre>
  <p>Both approaches have the advantage of not needing to read the entire file to memory so can be used on files of any size.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T23:40:45.067Z
Answers.Text: FWIW with GNU awk you could do `awk  'BEGIN{RS=">"} {ORS=RT} !/Sequence unavailable/' file` instead of piping to `sed`.
Answers.UserDisplayName: Ed Morton
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-05T13:08:12.123Z
Answers.LastActivityDate: 2022-11-05T13:08:12.123Z
Answers.UserDisplayName: terdon
Answers.Body: <p>Try this, if you have GNU <code>grep</code> (the default on Linux systems):</p>
  <pre><code>grep --no-group-separator -i -B 1 '^[ACGTN]\+$' input.fasta &gt; filtered.fasta
  </code></pre>
  <p>Explanation: find all lines that are composed of only A/C/G/T/N (case insensitive search), and show that line plus the preceding line. Don't put any breaks between matches that are not consecutive.</p>
  <p>This should work on both multi-line and single line fasta files.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-05T13:06:34.450Z
Answers.LastActivityDate: 2022-11-05T13:14:16.047Z
Answers.LastEditDate: 2022-11-05T13:14:16.047Z
Answers.UserDisplayName: gringer
Answers.Body: <p>If there are no line breaks within the sequence data (the format I use) then the following will work.</p>
  <pre><code>perl -0777 -nle 'while (/^(&gt;.+\n^[AGCT-]+)\n/gm){print $1,&quot;\n&quot;}' myfile.fa
  </code></pre>
  <hr />
  <p>Test</p>
  <pre><code>&gt;foo
  AGAGTAGATAGAAGATA
  &gt;bar
  AGAGATAGAGAGATATA
  &gt;baz
  Sequence unavailable
  </code></pre>
  <p>Output</p>
  <pre><code>&gt;foo
  AGAGTAGATAGAAGATA
  &gt;bar
  AGAGATAGAGAGATATA
  </code></pre>
  <hr />
  <p><strong>However</strong> If there are line breaks within the sequence data, please use the following ...</p>
  <pre><code>perl -0777 -nle 'while (/^(&gt;.+\n[AGCT-]{10}.+\n[^&gt;]+)/gm){print $1}' myfile.fa
  </code></pre>
  <p>Then pipe to a file,</p>
  <pre><code>perl -0777 -nle 'while (/^(&gt;.+\n[AGCT-]{10}.+\n[^&gt;]+)/gm){print $1}' myfile.fa &gt; myfile_minus_blanks.fa
  </code></pre>
  <p>This works due to a multi-line <code>m</code></p>
  <p>Test ...</p>
  <pre><code>&gt;foo
  AGAGTAGATAGAAGATA
  AGAGTAGATAGAAGATA
  AGAGTAGATAGAAGATA
  AGAGTAGATAGAAGATA
  &gt;bar
  AGAGATAGAGAGATATA
  AGAGATAGAGAGATATA
  &gt;baz
  Sequence unavailable
  </code></pre>
  <p>Out ...</p>
  <pre><code>&gt;foo
  AGAGTAGATAGAAGATA
  AGAGTAGATAGAAGATA
  AGAGTAGATAGAAGATA
  AGAGTAGATAGAAGATA
  &gt;bar
  AGAGATAGAGAGATATA
  AGAGATAGAGAGATATA
  </code></pre>
  <p><strong>Note</strong>
  If you've mixed bases, degenerate nucleotides,</p>
  <p><code>AGCT-</code> should be replaced with <code>AGCTRYWSMKNBDHV</code> (omitting <code>-</code> [not needed])</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-05T13:00:39.350Z
Answers.Text: Note that this will require loading the entire file into memory which might be an issue with large files.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-05T22:26:51.810Z
Answers.Text: I just processed a 22Gig file with a slightly more complicated command (removing loci). It worked fine.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-06T09:42:42.350Z
Answers.Text: Thank you very much for your kind response. This is work fro me.
Answers.UserDisplayName: Dr Sud
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-06T13:30:20.100Z
Answers.Text: Hi @DrSud great! I use it because its versatile (use it for different stuff). Please do remember to mark an answer "accepted" if you find it helpful and you can also now upvote all the answers given here.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-04T18:51:03.250Z
Answers.LastActivityDate: 2022-11-05T12:58:19.177Z
Answers.LastEditDate: 2022-11-05T12:58:19.177Z
Answers.UserDisplayName: M__
Answers.Body: <p>One way using <a href="/questions/tagged/sed" class="post-tag" title="show questions tagged &#39;sed&#39;" aria-label="show questions tagged &#39;sed&#39;" rel="tag" aria-labelledby="sed-container">sed</a>:</p>
  <pre><code>sed -e '/^&gt;/ { N; }; /Sequence unavailable/ { d; }' file.fa
  </code></pre>
  <p>For each header line, append the next line of input into the pattern space. Then, if the pattern space contains &quot;Sequence unavailable&quot; delete the pattern space and start a new cycle.</p>
  <hr />
  <p>Or using <a href="/questions/tagged/awk" class="post-tag" title="show questions tagged &#39;awk&#39;" aria-label="show questions tagged &#39;awk&#39;" rel="tag" aria-labelledby="awk-container">awk</a>:</p>
  <pre><code>awk '/^&gt;/ { f=$0; next } /^Sequence unavailable/ { next } f { print f; f=&quot;&quot; }1' file.fa
  </code></pre>
  <p>For each header line, use a variable to store the current record. We can use <a href="https://www.gnu.org/software/gawk/manual/html_node/Next-Statement.html" rel="nofollow noreferrer"><em>next</em></a> here to force awk to immediately stop processing the current record and move on to the next record, which is just the next line, since, by default, awk's <a href="https://www.gnu.org/software/gawk/manual/html_node/awk-split-records.html" rel="nofollow noreferrer"><em>record separator</em></a> is a newline character. Then, stop processing the current record and move on to the next record if that line begins with &quot;Sequence unavailable&quot;. Finally, print the header line if it exists and unset the variable that we used to store it if it does. Once we've printed the header, we can then just print the current record. We can do this by appending <em>1</em> to our command string. The <em>1</em> on the end just returns true, and by doing so, it performs the default action, which is to print the current record.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-05T18:22:51.380Z
Answers.Text: There's a bug in the sed code, `sed: 1: "/^>/ { N }; /Sequence u ...": extra characters at the end of N command`. OSX Darwin
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-05T21:54:48.893Z
Answers.Text: @M__ ah... my mistake. Should be good now I think. Thanks for testing it on OSX
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-05T22:29:39.493Z
Answers.Text: Thanks but same error but as I'm not feeling motivated to test on Linux ...  lets say it an OSX thing, most work on Linux. The awk works fine.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-06T14:20:15.060Z
Answers.Text: @M__ I don't think I read your error message properly before. Sorry. Hopefully the above will work now..
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-05T13:43:35.993Z
Answers.LastActivityDate: 2022-11-06T14:22:43.730Z
Answers.LastEditDate: 2022-11-06T14:22:43.730Z
Answers.UserDisplayName: Steve
Body: <p>I have a fasta file of 3'UTR sequences of the genes. Some of the gene do not have a 3'UTR sequence and reported with <code>sequence unavailable</code> but there is FASTA header.</p>
  <p>I need to delete or remove all the &quot;sequence unavailable&quot; lines along with their header, i.e. all blank fasta entries.</p>
  <p>Please suggest a command with <code>awk</code> or <code>sed</code> command.</p>
  <p><a href="https://i.sstatic.net/5orlh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5orlh.png" alt="Image of FASTA file" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-05T13:31:07.230Z
Comments.Text: Please [don't post images of text](https://bioinformatics.meta.stackexchange.com/q/326/298). Instead, copy/paste the text into your question and use the [formatting tools](https://bioinformatics.stackexchange.com/help/formatting) to format it as code. Also, what operating system are you using? We need to know so we can know what tools and what versions of the tools will be available.
Comments.UserDisplayName: terdon
Tags: <phylogenetics><phylogeny><fasta><awk><grep>
Title: Removing particular line along with Fasta header in fasta file
ThreadText: # Removing particular line along with Fasta header in fasta file
  
  (Asked by: username_0 on 2022-11-04)
  
  I have a fasta file of 3'UTR sequences of the genes. Some of the gene do not have a 3'UTR sequence and reported with `sequence unavailable` but there is FASTA header.
  
  I need to delete or remove all the "sequence unavailable" lines along with their header, i.e. all blank fasta entries.
  
  Please suggest a command with `awk` or `sed` command.
  
  [![Image of FASTA file](https://i.sstatic.net/5orlh.png)](https://i.sstatic.net/5orlh.png)
  
  **Tags:** phylogenetics, phylogeny, fasta, awk, grep
  
  > **Comments:**
  > **username_1**: Please [don't post images of text](https://bioinformatics.meta.stackexchange.com/q/326/298). Instead, copy/paste the text into your question and use the [formatting tools](https://bioinformatics.stackexchange.com/help/formatting) to format it as code. Also, what operating system are you using? We need to know so we can know what tools and what versions of the tools will be available.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2022-11-05. Score: 4
  
  You can use `awk` here, by setting the input (`RS`) and output (`ORS`) record separator, this is what defines a "line" in awk, to `>`. Then, tell it to print records that do not match `Sequence unavailable`. For instance, using this input file:
  
  ```
  $ cat file
  >foo
  AGAGTAGATAGAAGATA
  >bar
  AGAGATAGAGAGATATA
  >baz
  Sequence unavailable
  >bar1
  AGAGATAGAGAGATATA
  >asld
  Sequence unavailable
  >asd
  ACTTC
  
  ```
  
  You can do:
  
  ```
  $ awk  'BEGIN{RS=ORS=">"}!/Sequence unavailable/' file 
  >foo
  AGAGTAGATAGAAGATA
  >bar
  AGAGATAGAGAGATATA
  >bar1
  AGAGATAGAGAGATATA
  >asd
  ACTTC
  >
  
  ```
  
  That will add a trailing `>`, so you need one more step to remove it:
  
  ```
  $ awk  'BEGIN{RS=ORS=">"}!/Sequence unavailable/' file | sed 's/>$//'
  >foo
  AGAGTAGATAGAAGATA
  >bar
  AGAGATAGAGAGATATA
  >bar1
  AGAGATAGAGAGATATA
  >asd
  ACTTC
  
  ```
  
  ---
  
  Alternatively, using the `FastaToTbl` and `TblToFasta` scripts I have posted [here](https://bioinformatics.stackexchange.com/a/2651/298), you can do:
  
  ```
  $ FastaToTbl file | grep -v 'Sequence unavailable' | TblToFasta 
  >foo 
  AGAGTAGATAGAAGATA
  >bar 
  AGAGATAGAGAGATATA
  >bar1 
  AGAGATAGAGAGATATA
  >asd 
  ACTTC
  
  ```
  
  Both approaches have the advantage of not needing to read the entire file to memory so can be used on files of any size.
  
  > **Comments:**
  > **username_2**: FWIW with GNU awk you could do `awk  'BEGIN{RS=">"} {ORS=RT} !/Sequence unavailable/' file` instead of piping to `sed`.  
  
  ---
  
  ### Answer by username_3 on 2022-11-05. Score: 3
  
  Try this, if you have GNU `grep` (the default on Linux systems):
  
  ```
  grep --no-group-separator -i -B 1 '^[ACGTN]\+$' input.fasta > filtered.fasta
  
  ```
  
  Explanation: find all lines that are composed of only A/C/G/T/N (case insensitive search), and show that line plus the preceding line. Don't put any breaks between matches that are not consecutive.
  
  This should work on both multi-line and single line fasta files.
  
  
  ---
  
  ### Answer by username_4 on 2022-11-04. Score: 2
  
  If there are no line breaks within the sequence data (the format I use) then the following will work.
  
  ```
  perl -0777 -nle 'while (/^(>.+\n^[AGCT-]+)\n/gm){print $1,"\n"}' myfile.fa
  
  ```
  
  ---
  
  Test
  
  ```
  >foo
  AGAGTAGATAGAAGATA
  >bar
  AGAGATAGAGAGATATA
  >baz
  Sequence unavailable
  
  ```
  
  Output
  
  ```
  >foo
  AGAGTAGATAGAAGATA
  >bar
  AGAGATAGAGAGATATA
  
  ```
  
  ---
  
  **However** If there are line breaks within the sequence data, please use the following ...
  
  ```
  perl -0777 -nle 'while (/^(>.+\n[AGCT-]{10}.+\n[^>]+)/gm){print $1}' myfile.fa
  
  ```
  
  Then pipe to a file,
  
  ```
  perl -0777 -nle 'while (/^(>.+\n[AGCT-]{10}.+\n[^>]+)/gm){print $1}' myfile.fa > myfile_minus_blanks.fa
  
  ```
  
  This works due to a multi-line `m`
  
  Test ...
  
  ```
  >foo
  AGAGTAGATAGAAGATA
  AGAGTAGATAGAAGATA
  AGAGTAGATAGAAGATA
  AGAGTAGATAGAAGATA
  >bar
  AGAGATAGAGAGATATA
  AGAGATAGAGAGATATA
  >baz
  Sequence unavailable
  
  ```
  
  Out ...
  
  ```
  >foo
  AGAGTAGATAGAAGATA
  AGAGTAGATAGAAGATA
  AGAGTAGATAGAAGATA
  AGAGTAGATAGAAGATA
  >bar
  AGAGATAGAGAGATATA
  AGAGATAGAGAGATATA
  
  ```
  
  **Note**
  If you've mixed bases, degenerate nucleotides,
  
  `AGCT-` should be replaced with `AGCTRYWSMKNBDHV` (omitting `-` [not needed])
  
  > **Comments:**
  > **username_1**: Note that this will require loading the entire file into memory which might be an issue with large files.  
  > **username_4**: I just processed a 22Gig file with a slightly more complicated command (removing loci). It worked fine.  
  > **username_0**: Thank you very much for your kind response. This is work fro me.  
  > **username_4**: Hi @username_0 great! I use it because its versatile (use it for different stuff). Please do remember to mark an answer "accepted" if you find it helpful and you can also now upvote all the answers given here.  
  
  ---
  
  ### Answer by username_5 on 2022-11-05. Score: 2
  
  One way using [sed](/questions/tagged/sed "show questions tagged 'sed'"):
  
  ```
  sed -e '/^>/ { N; }; /Sequence unavailable/ { d; }' file.fa
  
  ```
  
  For each header line, append the next line of input into the pattern space. Then, if the pattern space contains "Sequence unavailable" delete the pattern space and start a new cycle.
  
  ---
  
  Or using [awk](/questions/tagged/awk "show questions tagged 'awk'"):
  
  ```
  awk '/^>/ { f=$0; next } /^Sequence unavailable/ { next } f { print f; f="" }1' file.fa
  
  ```
  
  For each header line, use a variable to store the current record. We can use [*next*](https://www.gnu.org/software/gawk/manual/html_node/Next-Statement.html) here to force awk to immediately stop processing the current record and move on to the next record, which is just the next line, since, by default, awk's [*record separator*](https://www.gnu.org/software/gawk/manual/html_node/awk-split-records.html) is a newline character. Then, stop processing the current record and move on to the next record if that line begins with "Sequence unavailable". Finally, print the header line if it exists and unset the variable that we used to store it if it does. Once we've printed the header, we can then just print the current record. We can do this by appending *1* to our command string. The *1* on the end just returns true, and by doing so, it performs the default action, which is to print the current record.
  
  > **Comments:**
  > **username_4**: There's a bug in the sed code, `sed: 1: "/^>/ { N }; /Sequence u ...": extra characters at the end of N command`. OSX Darwin  
  > **username_5**: @username_4 ah... my mistake. Should be good now I think. Thanks for testing it on OSX  
  > **username_4**: Thanks but same error but as I'm not feeling motivated to test on Linux ...  lets say it an OSX thing, most work on Linux. The awk works fine.  
  > **username_5**: @username_4 I don't think I read your error message properly before. Sorry. Hopefully the above will work now..  
  
  ---
  
  