Answers.Body: <p>You are getting a syntax error because your command is in double quotes. This means that the awk variables you are using, <code>$4</code> and <code>$5</code>, are being expanded by the shell before launching <code>awk</code>, and since they are not defined, they are passed to <code>awk</code> as empty, making your command:</p>
  <pre><code>awk -F'\t' -v OFS='\t' '/^[^#]/{-=5}{+=5}1' &quot;
  </code></pre>
  <p>And that is what gives you the syntax error. If you are 100% sure your file names contain no whitespace and no globbing characters, so you can use <code>ls</code> as you have done (<a href="https://mywiki.wooledge.org/ParsingLs" rel="nofollow noreferrer">which is generally a very bad idea</a>), then this should work:</p>
  <pre><code> ls ./*ratio.gff.gz | 
     parallel &quot;zcat {} | head | awk -F'\t' -v OFS='\t' '/^[^#]/{\\\$5-=5}{\\\$4+=5}1' &quot;
  </code></pre>
  <p>However, that won't change the original files and it isn't saving the output anywhere, so I doubt it's what you really want to do. This command will edit each file, and save the result into a new file with the same name plus <code>edited.gz</code>. So if you start with <code>foo.gff.gz</code>, you will get a new file called <code>foo.gff.gz.edited.gz</code>:</p>
  <pre><code>parallel &quot;awk -F'\t' -v OFS='\t' '/^[^#]/{\\\$5-=5}{\\\$4+=5}1' &lt;(zcat {}) | gzip &gt; {}.edited.gz&quot; ::: *gff.gz
  </code></pre>
  <p>Once you are sure this works as you want, you can rename all the <code>edited.gz</code> files back to the original name, overwriting the originals:</p>
  <pre><code>for f in *edited.gz; do mv -- &quot;$f&quot; &quot;${f//.edited.gz}&quot;; done
  </code></pre>
  <hr />
  <p><strong>IMPORTANT</strong>: The solution above is affected by a markdown rendering bug. There should only ever be <em>one</em> <code>\</code>, as can be seen in the source of this answer, and not three. Make sure to replace all occurrences of <code>\\\</code> in this answer with <code>\</code>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-07T21:00:26.090Z
Answers.Text: Thank you so much, this is exactly what I want to do. Somehow on the server I use (Debian) "\\\" also returns an error ```awk: 1: unexpected character '\' ```, if changed to "\", it worked.   ```parallel "awk -F'\t' -v OFS='\t' '/^[^#]/{\$5-=5}{\$4+=5}1' <(zcat {}) | gzip > {}.edited.gz" ::: *ratio.gff.gz```
Answers.UserDisplayName: ShannonTown
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-08T10:53:42.877Z
Answers.Text: Huh, sorry @ShannonTown that looks like a bug in the markdown rendering of the site. I only have one backslash in the answer (you can see if you try to edit it) but for some reason it is rendered as three!
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-07T12:23:24.743Z
Answers.LastActivityDate: 2024-08-08T17:38:51.293Z
Answers.LastEditDate: 2024-08-08T17:38:51.293Z
Answers.UserDisplayName: terdon
Body: <p>I have multiple gff files that I want to edit the start and end by 5 bp in batch. I found a relevant answer <a href="https://bioinformatics.stackexchange.com/questions/17715/edit-end-column-in-gff3-file">here</a> for single file, but I could not add comment there. The awk line works for my files if processing only one file, but when I try to do the same thing in batch, it returns syntax error. How should I do it with parallel?</p>
  <pre><code>&gt; ls ./*ratio.gff.gz | parallel &quot;zcat {} | head | awk -F'\t' -v OFS='\t' '/^[^#]/{<span class="math-container">$5-=5}{$</span>4+=5}1' &quot;
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  
  
  &gt; zcat xxxx_ratio.gff.gz | head | awk -F'\t' -v OFS='\t' '/^[^#]/{<span class="math-container">$5-=5}{$</span>4+=5}1'
  ##gff-version   3               5
  # biweight_mean=-0.22621330627900907                    5
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5136    5179    0.60    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005131;count=2
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5181    5229    0.65    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005176;count=1
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5241    5287    0.41    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005236;count=1
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5286    5327    0.09    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005281;count=1
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5406    5452    0.92    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005401;count=1
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5466    5519    0.98    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005461;count=1
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5506    5559    0.26    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005501;count=1
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5576    5628    0.41    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005571;count=1
  </code></pre>
  <p>Thank you!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-06T23:48:54.713Z
Comments.Text: what happens if you move the awk command to a shell script? I suspect that the quoting is messing it up somehow.
Comments.UserDisplayName: Maximilian Press
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-07T12:02:33.240Z
Comments.Text: [_Never_ do `ls | command`](https://mywiki.wooledge.org/ParsingLs), that is asking for trouble. To be fair, it isn't causing issues here, but it is still a bad habit.
Comments.UserDisplayName: terdon
Tags: <awk><parallel>
Title: Edit column in multiple GFF3 file using parallel and awk?
ThreadText: # Edit column in multiple GFF3 file using parallel and awk?
  
  (Asked by: username_0 on 2024-08-06)
  
  I have multiple gff files that I want to edit the start and end by 5 bp in batch. I found a relevant answer [here](https://bioinformatics.stackexchange.com/questions/17715/edit-end-column-in-gff3-file) for single file, but I could not add comment there. The awk line works for my files if processing only one file, but when I try to do the same thing in batch, it returns syntax error. How should I do it with parallel?
  
  ```
  > ls ./*ratio.gff.gz | parallel "zcat {} | head | awk -F'\t' -v OFS='\t' '/^[^#]/{$5-=5}{$4+=5}1' "
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  awk: line 1: syntax error at or near -=
  awk: line 1: syntax error at or near +=
  
  > zcat xxxx_ratio.gff.gz | head | awk -F'\t' -v OFS='\t' '/^[^#]/{$5-=5}{$4+=5}1'
  ##gff-version   3               5
  # biweight_mean=-0.22621330627900907                    5
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5136    5179    0.60    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005131;count=2
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5181    5229    0.65    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005176;count=1
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5241    5287    0.41    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005236;count=1
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5286    5327    0.09    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005281;count=1
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5406    5452    0.92    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005401;count=1
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5466    5519    0.98    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005461;count=1
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5506    5559    0.26    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005501;count=1
  chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5576    5628    0.41    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005571;count=1
  
  ```
  
  Thank you!
  
  **Tags:** awk, parallel
  
  > **Comments:**
  > **username_1**: what happens if you move the awk command to a shell script? I suspect that the quoting is messing it up somehow.  
  > **username_2**: [_Never_ do `ls | command`](https://mywiki.wooledge.org/ParsingLs), that is asking for trouble. To be fair, it isn't causing issues here, but it is still a bad habit.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-08-07. Score: 2 | Accepted answer
  
  You are getting a syntax error because your command is in double quotes. This means that the awk variables you are using, `$4` and `$5`, are being expanded by the shell before launching `awk`, and since they are not defined, they are passed to `awk` as empty, making your command:
  
  ```
  awk -F'\t' -v OFS='\t' '/^[^#]/{-=5}{+=5}1' "
  
  ```
  
  And that is what gives you the syntax error. If you are 100% sure your file names contain no whitespace and no globbing characters, so you can use `ls` as you have done ([which is generally a very bad idea](https://mywiki.wooledge.org/ParsingLs)), then this should work:
  
  ```
   ls ./*ratio.gff.gz | 
     parallel "zcat {} | head | awk -F'\t' -v OFS='\t' '/^[^#]/{\\\$5-=5}{\\\$4+=5}1' "
  
  ```
  
  However, that won't change the original files and it isn't saving the output anywhere, so I doubt it's what you really want to do. This command will edit each file, and save the result into a new file with the same name plus `edited.gz`. So if you start with `foo.gff.gz`, you will get a new file called `foo.gff.gz.edited.gz`:
  
  ```
  parallel "awk -F'\t' -v OFS='\t' '/^[^#]/{\\\$5-=5}{\\\$4+=5}1' <(zcat {}) | gzip > {}.edited.gz" ::: *gff.gz
  
  ```
  
  Once you are sure this works as you want, you can rename all the `edited.gz` files back to the original name, overwriting the originals:
  
  ```
  for f in *edited.gz; do mv -- "$f" "${f//.edited.gz}"; done
  
  ```
  
  ---
  
  **IMPORTANT**: The solution above is affected by a markdown rendering bug. There should only ever be *one* `\`, as can be seen in the source of this answer, and not three. Make sure to replace all occurrences of `\\\` in this answer with `\`.
  
  > **Comments:**
  > **username_0**: Thank you so much, this is exactly what I want to do. Somehow on the server I use (Debian) "\\\" also returns an error ```awk: 1: unexpected character '\' ```, if changed to "\", it worked.   ```parallel "awk -F'\t' -v OFS='\t' '/^[^#]/{\$5-=5}{\$4+=5}1' <(zcat {}) | gzip > {}.edited.gz" ::: *ratio.gff.gz```  
  > **username_2**: Huh, sorry @username_0 that looks like a bug in the markdown rendering of the site. I only have one backslash in the answer (you can see if you try to edit it) but for some reason it is rendered as three!  
  
  ---
  
  