Answers.Body: <p>Using a <code>for</code> loop for this is <a href="https://mywiki.wooledge.org/ParsingLs" rel="nofollow noreferrer">by far the safer option</a>. However, you need to <a href="https://unix.stackexchange.com/a/131767/22222">quote your variables</a> and also make sure you're passing the right variable name. While you're at it, you can also remove the extension so that your filtered file name can be <code>foo_out.vcf</code> instead of <code>foo.vcf_out.vcf</code> (which is what your loop would create. Putting all that together:</p>
  
  <pre><code>for f in *.vcf; do 
      echo "filtering <span class="math-container">$f"; 
  bcftools filter -i 'QUAL &gt; 1000' -o filter/"$</span>{f//.vcf}"_out.vcf "$f"; done
  fi
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-03T19:52:24.207Z
Answers.Text: I like the "${f//.vcf}", thanks - my filenames have been growing unnecessarily long!
Answers.UserDisplayName: Joanne
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-03T16:12:51.863Z
Answers.LastActivityDate: 2018-12-03T16:12:51.863Z
Answers.UserDisplayName: terdon
Answers.Body: <p>For one-liners, <code>xargs</code> is often more convenient than the <code>for</code> loop:</p>
  
  <pre class="lang-shell prettyprint-override"><code>ls *.vcf | sed s,.vcf$,, \
    | xargs -i echo 'bcftools filter -i "QUAL &gt; 1000" -o "filter/{}_out.vcf" "{}.vcf"' \
    | parallel
  </code></pre>
  
  <p>Note that your command line (with @WouterDeCoster's fix) writes output to <code>foo.vcf_out.vcf</code>. My command line writes to <code>foo_out.vcf</code>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-03T16:17:40.440Z
Answers.Text: Xargs is indeed convenient, but parsing `ls` is very fragile and will break on any non-standard file name. Why not `for f in *vcf; do printf 'bcftools filter -i "QUAL > 1000" -o filter/"%s" "%s"\n' "${f//.vcf}_out.vcf" "$f"; done | parallel`? That seems as simple as `xargs` but can deal with arbitrary file names.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-03T17:19:39.100Z
Answers.Text: @terdon I don't see that is an issue. You can simply add quotation marks.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-03T17:27:49.653Z
Answers.Text: Yes, that will protect from spaces, but not from newlines. And OK, having newlines in filenames isn't something that happens very often but it is possible. I freely admit, however, that I may just be nitpicking because I'm too used to hanging out on [unix.se] where this sort of thing is considered very important and any example of parsing `ls` is met with horror. Still, since the shell loop is safe and painless, it just seems better to use that instead.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-02T02:56:35.130Z
Answers.LastActivityDate: 2018-12-03T17:18:56.133Z
Answers.LastEditDate: 2018-12-03T17:18:56.133Z
Answers.UserDisplayName: user172818
Answers.Body: <p>I assume your shell is looking for the variable <code>$f_out</code>, but can't find it because you did not define it. It probably generated a hidden file with the name <code>.vcf</code> in your output directory.</p>
  
  <p>Instead, use the following: <code>-o <span class="math-container">${f}_out.vcf</code>. This will "protect/limit" the bash variable to just <code>$</span>f</code> rather than <code>$f_out</code>.</p>
  
  <p>When encountering issues like this it's worth putting <code>echo</code> in your loop right before your command, in this case bcftools (adding quotes whenever necessary). That would show you the command as how it is going to be executed and would show you, in this case, that the output name is incorrect.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-03T17:44:09.363Z
Answers.Text: This works thanks! But I wonder why I've had a similar loop working fine with `$f.SnpSift.vcf` as my output? Does it understand the `.` differently than the `_` after `$f`?
Answers.UserDisplayName: Joanne
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-03T19:14:24.803Z
Answers.Text: Yes that's right, and easy to test using a dummy variable and `echo`
Answers.UserDisplayName: Wouter De Coster
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-06T09:14:55.877Z
Answers.Text: Alternatively, instead of (or in addition to) `echo`-ing you can use `set -u` which will give an error on undefined variables (this assumes you use Bash or Zsh as your shell (not sure about others)).
Answers.UserDisplayName: ph0t0nix
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-01T09:57:56.710Z
Answers.LastActivityDate: 2018-12-01T09:57:56.710Z
Answers.UserDisplayName: Wouter De Coster
Body: <p>Probably a silly oversight on my part, but I'm trying to filter all the vcfs in a directory with bcftools using a simple loop. My basic command is working fine:</p>
  
  <pre><code>bcftools filter -i 'QUAL &gt; 1000' -o filter/file1out.vcf file1in.vcf 
  </code></pre>
  
  <p>but when I try to loop it, it echoes through each filename, but generates no output files or error messages.</p>
  
  <pre><code>for f in *.vcf; do echo "filtering <span class="math-container">$f"; bcftools filter -i 'QUAL &gt; 1000' -o filter/$</span>f_out.vcf $f; done
  </code></pre>
  
  <p>Am I not handling the output direction properly? Any help much appreciated!</p>
  
Tags: <vcf><bcftools>
Title: bcftools filtering all files in a directory
ThreadText: # bcftools filtering all files in a directory
  
  (Asked by: username_0 on 2018-11-30)
  
  Probably a silly oversight on my part, but I'm trying to filter all the vcfs in a directory with bcftools using a simple loop. My basic command is working fine:
  
  ```
  bcftools filter -i 'QUAL > 1000' -o filter/file1out.vcf file1in.vcf 
  
  ```
  
  but when I try to loop it, it echoes through each filename, but generates no output files or error messages.
  
  ```
  for f in *.vcf; do echo "filtering $f"; bcftools filter -i 'QUAL > 1000' -o filter/$f_out.vcf $f; done
  
  ```
  
  Am I not handling the output direction properly? Any help much appreciated!
  
  **Tags:** vcf, bcftools
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2018-12-01. Score: 5 | Accepted answer
  
  I assume your shell is looking for the variable `$f_out`, but can't find it because you did not define it. It probably generated a hidden file with the name `.vcf` in your output directory.
  
  Instead, use the following: `-o ${f}_out.vcf`. This will "protect/limit" the bash variable to just `$f` rather than `$f_out`.
  
  When encountering issues like this it's worth putting `echo` in your loop right before your command, in this case bcftools (adding quotes whenever necessary). That would show you the command as how it is going to be executed and would show you, in this case, that the output name is incorrect.
  
  > **Comments:**
  > **username_0**: This works thanks! But I wonder why I've had a similar loop working fine with `$f.SnpSift.vcf` as my output? Does it understand the `.` differently than the `_` after `$f`?  
  > **username_3**: Yes that's right, and easy to test using a dummy variable and `echo`  
  > **username_4**: Alternatively, instead of (or in addition to) `echo`-ing you can use `set -u` which will give an error on undefined variables (this assumes you use Bash or Zsh as your shell (not sure about others)).  
  
  ---
  
  ### Answer by username_1 on 2018-12-03. Score: 2
  
  Using a `for` loop for this is [by far the safer option](https://mywiki.wooledge.org/ParsingLs). However, you need to [quote your variables](https://unix.stackexchange.com/a/131767/22222) and also make sure you're passing the right variable name. While you're at it, you can also remove the extension so that your filtered file name can be `foo_out.vcf` instead of `foo.vcf_out.vcf` (which is what your loop would create. Putting all that together:
  
  ```
  for f in *.vcf; do 
      echo "filtering $f"; 
  bcftools filter -i 'QUAL > 1000' -o filter/"${f//.vcf}"_out.vcf "$f"; done
  fi
  
  ```
  
  > **Comments:**
  > **username_0**: I like the "${f//.vcf}", thanks - my filenames have been growing unnecessarily long!  
  
  ---
  
  ### Answer by username_2 on 2018-12-02. Score: 1
  
  For one-liners, `xargs` is often more convenient than the `for` loop:
  
  ```shell
  ls *.vcf | sed s,.vcf$,, \
    | xargs -i echo 'bcftools filter -i "QUAL > 1000" -o "filter/{}_out.vcf" "{}.vcf"' \
    | parallel
  
  ```
  
  Note that your command line (with @username_3's fix) writes output to `foo.vcf_out.vcf`. My command line writes to `foo_out.vcf`.
  
  > **Comments:**
  > **username_1**: Xargs is indeed convenient, but parsing `ls` is very fragile and will break on any non-standard file name. Why not `for f in *vcf; do printf 'bcftools filter -i "QUAL > 1000" -o filter/"%s" "%s"\n' "${f//.vcf}_out.vcf" "$f"; done | parallel`? That seems as simple as `xargs` but can deal with arbitrary file names.  
  > **username_2**: @username_1 I don't see that is an issue. You can simply add quotation marks.  
  > **username_1**: Yes, that will protect from spaces, but not from newlines. And OK, having newlines in filenames isn't something that happens very often but it is possible. I freely admit, however, that I may just be nitpicking because I'm too used to hanging out on [unix.se] where this sort of thing is considered very important and any example of parsing `ls` is met with horror. Still, since the shell loop is safe and painless, it just seems better to use that instead.  
  
  ---
  
  