Answers.Body: <p>Your awk command is inside double quotes (<code>""</code>) which means that all of the awk field variables (e.g. <code>$3</code>) are being interpreted by the shell. Since <code>$3</code> is not set in the shell, your awk is trying to compare an empty string and that's why you get the error:</p>
  
  <pre><code>awk: cmd. line:1: {if( != chrM &amp;&amp;  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:      ^ syntax error
  </code></pre>
  
  <p>Note how the error message shows <code>if( != chrM</code> and not <code>if($3 != chrM</code>. Since you <em>need</em> to use double quotes here because you need both an outer (for <code>parallel</code>) and an inner (for awk) quote, and single quotes can't be nested, you will have to escape the awk variables:</p>
  
  <pre><code>ls *.bam | parallel "samtools view -h {} | 
      awk '{if(\$3 != "chrM" &amp;&amp; \$3 != "chrUn"){print \$0}}' | samtools view -Shb {.}.bam" 
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-16T11:01:44.597Z
Answers.Text: looks like i need to clear my fundamental  awk concepts let me run and will update you. now i ran your code i get output like this "�&Ct�wh���x�Y#��J�f���{/.i��7��%M`�������`C�̛�<kbb�������B��f����<#?���
                                                                                          :P�y���
  u��O��r.�_2ؑ�xu�zl�,xϭ"2�*�Uo�ȩ�@ayi���
  /0���/���۶�
                 �>�Xh� ��Ί�5h�f~���euSa1����;5��{���R�3L����p%��[<�1)M�y!�w��#���_�`�V5rj��v�}�
  S-���y)�Nμ4��Jӕ�D���*n�1��ޥ�g�����y���䒋�N��)���<:���mʲ�溎[�ʘ���K��[punit@localhost chrm]$ 62;c62;c62;c62;c^C
  "
Answers.UserDisplayName: kcm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-16T11:12:02.113Z
Answers.Text: @krushnachChandra well yes, of course. You are printing a bam file which is compressed binary data. What were you expecting to see? If you want sam output, remove the `b` from the options you give to `samtools view`. The `b` makes it print bam output.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-16T11:14:58.687Z
Answers.Text: so i idea is after filtering the mitochondrial read and unassembled one i want  bam output written to file with bam format which will be using for downstream analysis.I don't want to print it on the terminal . tried it with -o to write the output seems like i couldn;t make it work..
Answers.UserDisplayName: kcm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-16T11:19:00.160Z
Answers.Text: @krushnachChandra `-o` should work. Also, you could redirect in the parallel command: `ls *.bam | parallel "samtools view -h {} | awk '{if(\$3 != "chrM" && \$3 != "chrUn"){print \$0}}' | samtools view -Shb  > {}.bam"`. But as I said under your question, the new error is completely unrelated. You probably have a malformed bam file. But please ask a new question about that. Even better, run `for bam in *bam; do samtools view -h "$bam" | head -n1 || echo "$bam file is corrupted"; done` That should help you find the bad file.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-16T11:22:23.410Z
Answers.Text: will run it and see
Answers.UserDisplayName: kcm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-16T12:31:23.617Z
Answers.Text: its working ,the output is written to bam file
Answers.UserDisplayName: kcm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-16T07:03:38.737Z
Answers.Text: i have accepted the answer sorry i missed it to mark since i saw this thread active again .
Answers.UserDisplayName: kcm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-16T10:54:04.467Z
Answers.LastActivityDate: 2019-04-16T10:54:04.467Z
Answers.UserDisplayName: terdon
Body: <p>This is the one i used for a single bam file to filter its mitochondrial as well as unassembled read</p>
  <pre><code>samtools view -h SRR2920466.bam | awk '{if(<span class="math-container">$3!= "chrM" &amp;&amp; $</span>3 != &quot;chrUn&quot;){print $0}}' | samtools view -Shb - &gt; this.filter.bam 
  </code></pre>
  <p>Is there any way i can do it on multiple files with parallel as i do to index and sort bam files using samtools .</p>
  <p>So i tried this ,which is of course not working as its not able to pipe properly because it shows me error regarding additional fonts</p>
  <pre><code>ls *.bam | parallel &quot;samtools view -h {} | awk '{if(<span class="math-container">$3 != "chrM" &amp;&amp; $</span>3 != &quot;chrUn&quot;){print $0}}'| samtools view -Shb {.}.bam&quot; 
  </code></pre>
  <p>Any help with my above codes? As i get it there is wrong with my pipe .</p>
  <p>After updating the samtools to version 1.9 i ran the small script</p>
  <p>i get this error</p>
  <pre><code>awk: cmd. line:1: {if( != chrM &amp;&amp;  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:      ^ syntax error
  awk: cmd. line:1: {if( != chrM &amp;&amp;  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:                                    ^ unterminated regexp
  awk: cmd. line:1: {if( != chrM &amp;&amp;  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:                                                   ^ unexpected newline or end of string
  awk: cmd. line:1: {if( != chrM &amp;&amp;  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:      ^ syntax error
  awk: cmd. line:1: {if( != chrM &amp;&amp;  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:                                    ^ unterminated regexp
  awk: cmd. line:1: {if( != chrM &amp;&amp;  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:                                                   ^ unexpected newline or end of string
  awk: cmd. line:1: {if( != chrM &amp;&amp;  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:      ^ syntax error
  awk: cmd. line:1: {if( != chrM &amp;&amp;  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:                                    ^ unterminated regexp
  awk: cmd. line:1: {if( != chrM &amp;&amp;  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:                                                   ^ unexpected newline or end of string
  </code></pre>
  <p>I have 30 bam files i tried with 2 to see if its working or not , after the suggestion of @WouterDeCoster i tried the above one after updating samtools . the basic samtool with egrep works but for multiple file i couldn;t do it.</p>
  
Tags: <awk><shell>
Title: removing mitochondrial read and unassembled "random" from multiple bam files
ThreadText: # removing mitochondrial read and unassembled "random" from multiple bam files
  
  (Asked by: username_0 on 2019-04-16)
  
  This is the one i used for a single bam file to filter its mitochondrial as well as unassembled read
  
  ```
  samtools view -h SRR2920466.bam | awk '{if($3!= "chrM" && $3 != "chrUn"){print $0}}' | samtools view -Shb - > this.filter.bam 
  
  ```
  
  Is there any way i can do it on multiple files with parallel as i do to index and sort bam files using samtools .
  
  So i tried this ,which is of course not working as its not able to pipe properly because it shows me error regarding additional fonts
  
  ```
  ls *.bam | parallel "samtools view -h {} | awk '{if($3 != "chrM" && $3 != "chrUn"){print $0}}'| samtools view -Shb {.}.bam" 
  
  ```
  
  Any help with my above codes? As i get it there is wrong with my pipe .
  
  After updating the samtools to version 1.9 i ran the small script
  
  i get this error
  
  ```
  awk: cmd. line:1: {if( != chrM &&  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:      ^ syntax error
  awk: cmd. line:1: {if( != chrM &&  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:                                    ^ unterminated regexp
  awk: cmd. line:1: {if( != chrM &&  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:                                                   ^ unexpected newline or end of string
  awk: cmd. line:1: {if( != chrM &&  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:      ^ syntax error
  awk: cmd. line:1: {if( != chrM &&  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:                                    ^ unterminated regexp
  awk: cmd. line:1: {if( != chrM &&  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:                                                   ^ unexpected newline or end of string
  awk: cmd. line:1: {if( != chrM &&  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:      ^ syntax error
  awk: cmd. line:1: {if( != chrM &&  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:                                    ^ unterminated regexp
  awk: cmd. line:1: {if( != chrM &&  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:                                                   ^ unexpected newline or end of string
  
  ```
  
  I have 30 bam files i tried with 2 to see if its working or not , after the suggestion of @WouterDeCoster i tried the above one after updating samtools . the basic samtool with egrep works but for multiple file i couldn;t do it.
  
  **Tags:** awk, shell
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-04-16. Score: 1 | Accepted answer
  
  Your awk command is inside double quotes (`""`) which means that all of the awk field variables (e.g. `$3`) are being interpreted by the shell. Since `$3` is not set in the shell, your awk is trying to compare an empty string and that's why you get the error:
  
  ```
  awk: cmd. line:1: {if( != chrM &&  != chrUn){print ./index_sort.sh}}
  awk: cmd. line:1:      ^ syntax error
  
  ```
  
  Note how the error message shows `if( != chrM` and not `if($3 != chrM`. Since you *need* to use double quotes here because you need both an outer (for `parallel`) and an inner (for awk) quote, and single quotes can't be nested, you will have to escape the awk variables:
  
  ```
  ls *.bam | parallel "samtools view -h {} | 
      awk '{if(\$3 != "chrM" && \$3 != "chrUn"){print \$0}}' | samtools view -Shb {.}.bam" 
  
  ```
  
  > **Comments:**
  > **username_0**: looks like i need to clear my fundamental  awk concepts let me run and will update you. now i ran your code i get output like this "�&Ct�wh���x�Y#��J�f���{/.i��7��%M`�������`C�̛�<kbb�������B��f����<#?���                                                                                         :P�y��� u��O��r.�_2ؑ�xu�zl�,xϭ"2�*�Uo�ȩ�@ayi��� /0���/���۶�                �>�Xh� ��Ί�5h�f~���euSa1����;5��{���R�3L����p%��[<�1)M�y!�w��#���_�`�V5rj��v�}� S-���y)�Nμ4��Jӕ�D���*n�1��ޥ�g�����y���䒋�N��)���<:���mʲ�溎[�ʘ���K��[punit@localhost chrm]$ 62;c62;c62;c62;c^C "  
  > **username_1**: @krushnachChandra well yes, of course. You are printing a bam file which is compressed binary data. What were you expecting to see? If you want sam output, remove the `b` from the options you give to `samtools view`. The `b` makes it print bam output.  
  > **username_0**: so i idea is after filtering the mitochondrial read and unassembled one i want  bam output written to file with bam format which will be using for downstream analysis.I don't want to print it on the terminal . tried it with -o to write the output seems like i couldn;t make it work..  
  > **username_1**: @krushnachChandra `-o` should work. Also, you could redirect in the parallel command: `ls *.bam | parallel "samtools view -h {} | awk '{if(\$3 != "chrM" && \$3 != "chrUn"){print \$0}}' | samtools view -Shb  > {}.bam"`. But as I said under your question, the new error is completely unrelated. You probably have a malformed bam file. But please ask a new question about that. Even better, run `for bam in *bam; do samtools view -h "$bam" | head -n1 || echo "$bam file is corrupted"; done` That should help you find the bad file.  
  > **username_0**: will run it and see  
  > **username_0**: its working ,the output is written to bam file  
  > **username_0**: i have accepted the answer sorry i missed it to mark since i saw this thread active again .  
  
  ---
  
  