Body: <p>The <code>-e</code> and <code>-i</code> options of the <code>bcftools filter</code> command appear, by default, to only allow for including or excluding <em>sites</em>.  E.g., <code>-e 'FMT/DP &lt; 10'</code> removes sites where any sample has <code>DP &lt; 10</code>, and <code>-e 'MEAN(FMT/DP) &lt; 10'</code> removes sites where average depth across samples is &lt; 10.</p>
  <p>I would like to perform effectively similar filtering commands, but in a way that includes or excludes samples, instead of sites.  E.g., exclude all <em>samples</em> that have <code>DP &lt; 10</code> at any <em>site</em>, or exclude all samples with an average depth across all sites &lt; 10, or exclude all samples that have a genotype quality of &lt; 20 at 10% of genotyped sites.</p>
  <p>Such commands can very easily be written to apply to sites, but there does not seem to be a straightforward way of applying such filtering expressions to samples in <code>bcftools</code>.</p>
  <p>One option mentioned in the manual is to use the double <code>||</code> operator, as described in the <a href="http://samtools.github.io/bcftools/bcftools.html#expressions" rel="nofollow noreferrer">man page</a>.  As described in my <a href="https://github.com/samtools/bcftools/issues/1384#issuecomment-770082770" rel="nofollow noreferrer">comment</a> on a Github issue, however, this does not seem to work as a filtering operation for including or excluding entire columns in a VCF file, but rather only for selecting more or less individual records (e.g., as illustrated in the example on <a href="http://samtools.github.io/bcftools/howtos/filtering.html" rel="nofollow noreferrer">this tutorial page</a>).  In addition, it would seem limited in functionality (e.g., functions such as <code>MEAN(FMT/DP) &lt; 10</code> would still be evaluated across samples, not sites, when combined with a <code>||</code> operator).</p>
  <p>The only other method mentioned in the manual is to use the <code>SMPL</code> or <code>s</code> prefix to functions which evaluate <code>FORMAT</code> fields in expressions passed to <code>filter</code>.  This also does not appear to work as I would expect.  I'm not sure I understand what it <em>is</em> doing, but it definitely <em>isn't</em> removing columns from the VCF file.  Instead it seems to be using as a metric calculated across sites on a per sample basis as a filtering criteria applied to sites (but I could just be confused):</p>
  <pre><code>nSites_unfiltered=<span class="math-container">$(bcftools view -H unfiltered.vcf.gz | wc -l) 
  nSamples_unfiltered=$</span>(bcftools query unfiltered.bcf.gz -l | wc -l)
  
  nSites_DPfilter=<span class="math-container">$(bcftools filter unfiltered.bcf.gz -e 'sMEAN(FORMAT/DP)&lt;5' | bcftools view -H | wc -l)
  nSamples_DPfilter=$</span>(bcftools filter unfiltered.bcf.gz -e 'sMEAN(FORMAT/DP)&lt;5' | bcftools query -l | wc -l)
  
  echo &quot;scale=5; <span class="math-container">$nSites_DPfilter / $</span>nSites_unfiltered&quot; | bc #0.17349 - sites are removed, even though the filter is being applied to samples
  echo &quot;scale=5; <span class="math-container">$nSamples_DPfilter / $</span>nSamples_unfiltered&quot; | bc #1.00000 - no samples removed, despite, again, the filter being applied to samples
  
  nSites_normalDPFilter=<span class="math-container">$(bcftools filter unfiltered.bcf.gz -e 'MEAN(FMT/DP) &lt; 5' | bcftools view -H | wc -l)
  echo "scale=5; $</span>nSites_normalDPFilter / $nSites_unfiltered&quot; | bc #0.83121 - fewer sites removed than the sMEAN filter
  </code></pre>
  <p>The <code>.bcf.gz</code> passed to these commands is here: <a href="https://www.dropbox.com/s/fwj1snabsb763sn/unfiltered.bcf.gz?dl=0" rel="nofollow noreferrer">https://www.dropbox.com/s/fwj1snabsb763sn/unfiltered.bcf.gz?dl=0</a>
  Its <code>.csi</code> index file is here: <a href="https://www.dropbox.com/s/jzdq6y91nkencmy/unfiltered.bcf.gz.csi?dl=0" rel="nofollow noreferrer">https://www.dropbox.com/s/jzdq6y91nkencmy/unfiltered.bcf.gz.csi?dl=0</a></p>
  <p>In any event, the only option I have been able to come up with that actually excludes or includes entire sample IDs is to extract summary statistics using <code>bcftools stats</code>, export the desired statistics from the <code>PSC, Per-sample counts</code> table, and perform my own exploratory analysis outside of <code>bcftools</code>.  Then I can use <code>bcftools view -S</code> to explicitly remove samples by their ID, which I have identified manually as failing some thresholding criteria.  This also is obviously cumbersome, and only works for <code>FMT</code> fields that are included in <code>stats</code> (e.g., not <code>GQ</code> or <code>PL</code>).</p>
  <p>I am not sure if I am missing some basic functionality of <code>bcftools</code>, or if this is more of a feature request.  Any information would be appreciated.</p>
  <p>Thanks!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-02T18:30:33.340Z
Comments.Text: You may want to take a look at vcftools' genotype filtering options or the custom JEXL expressions that GATK's SelectVariants allows. Are you sure your sample-filtering operations won't produce a variable number of columns for each row? Also, remember that `bcftools filter` will not remove samples no matter what you try. It's a utility to filter sites, and it can only filter sites.
Comments.UserDisplayName: Ram RS
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-03T21:34:51.170Z
Comments.Text: Thanks very much.  Since every site has genotypes for every sample (if you include missing genotype entries), I'm not sure I understand how removing certain columns in their entirety would create any problems in terms of uneven dimensions.  Indeed, this is the function of `bcftools view -s ^<list.of.sample.IDs>`, if I am not mistaken.  `vcftools` also only allows for filtering samples based on ID.  Thank you for the reference to SelectVariants.  Do you have a link to a manual, or example code to address any of the example use-cases referenced in my question?  Thanks!
Comments.UserDisplayName: shbrainard
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-04T21:00:43.920Z
Comments.Text: Not really, no. I have rarely ever used SelectVariants when it wasn't already implemented in a pipeline (GATK tools can be a pain to use at times). If push comes to shove, it might be worth converting the VCF to MAF format so the data is at individual level.
Comments.UserDisplayName: Ram RS
Tags: <vcf><snp><filtering><bcftools>
Title: Filtering samples by FORMAT attributes using bcftools
ThreadText: # Filtering samples by FORMAT attributes using bcftools
  
  (Asked by: username_0 on 2021-02-02)
  
  The `-e` and `-i` options of the `bcftools filter` command appear, by default, to only allow for including or excluding *sites*. E.g., `-e 'FMT/DP < 10'` removes sites where any sample has `DP < 10`, and `-e 'MEAN(FMT/DP) < 10'` removes sites where average depth across samples is < 10.
  
  I would like to perform effectively similar filtering commands, but in a way that includes or excludes samples, instead of sites. E.g., exclude all *samples* that have `DP < 10` at any *site*, or exclude all samples with an average depth across all sites < 10, or exclude all samples that have a genotype quality of < 20 at 10% of genotyped sites.
  
  Such commands can very easily be written to apply to sites, but there does not seem to be a straightforward way of applying such filtering expressions to samples in `bcftools`.
  
  One option mentioned in the manual is to use the double `||` operator, as described in the [man page](http://samtools.github.io/bcftools/bcftools.html#expressions). As described in my [comment](https://github.com/samtools/bcftools/issues/1384#issuecomment-770082770) on a Github issue, however, this does not seem to work as a filtering operation for including or excluding entire columns in a VCF file, but rather only for selecting more or less individual records (e.g., as illustrated in the example on [this tutorial page](http://samtools.github.io/bcftools/howtos/filtering.html)). In addition, it would seem limited in functionality (e.g., functions such as `MEAN(FMT/DP) < 10` would still be evaluated across samples, not sites, when combined with a `||` operator).
  
  The only other method mentioned in the manual is to use the `SMPL` or `s` prefix to functions which evaluate `FORMAT` fields in expressions passed to `filter`. This also does not appear to work as I would expect. I'm not sure I understand what it *is* doing, but it definitely *isn't* removing columns from the VCF file. Instead it seems to be using as a metric calculated across sites on a per sample basis as a filtering criteria applied to sites (but I could just be confused):
  
  ```
  nSites_unfiltered=$(bcftools view -H unfiltered.vcf.gz | wc -l) 
  nSamples_unfiltered=$(bcftools query unfiltered.bcf.gz -l | wc -l)
  
  nSites_DPfilter=$(bcftools filter unfiltered.bcf.gz -e 'sMEAN(FORMAT/DP)<5' | bcftools view -H | wc -l)
  nSamples_DPfilter=$(bcftools filter unfiltered.bcf.gz -e 'sMEAN(FORMAT/DP)<5' | bcftools query -l | wc -l)
  
  echo "scale=5; $nSites_DPfilter / $nSites_unfiltered" | bc #0.17349 - sites are removed, even though the filter is being applied to samples
  echo "scale=5; $nSamples_DPfilter / $nSamples_unfiltered" | bc #1.00000 - no samples removed, despite, again, the filter being applied to samples
  
  nSites_normalDPFilter=$(bcftools filter unfiltered.bcf.gz -e 'MEAN(FMT/DP) < 5' | bcftools view -H | wc -l)
  echo "scale=5; $nSites_normalDPFilter / $nSites_unfiltered" | bc #0.83121 - fewer sites removed than the sMEAN filter
  
  ```
  
  The `.bcf.gz` passed to these commands is here: <https://www.dropbox.com/s/fwj1snabsb763sn/unfiltered.bcf.gz?dl=0>
  Its `.csi` index file is here: <https://www.dropbox.com/s/jzdq6y91nkencmy/unfiltered.bcf.gz.csi?dl=0>
  
  In any event, the only option I have been able to come up with that actually excludes or includes entire sample IDs is to extract summary statistics using `bcftools stats`, export the desired statistics from the `PSC, Per-sample counts` table, and perform my own exploratory analysis outside of `bcftools`. Then I can use `bcftools view -S` to explicitly remove samples by their ID, which I have identified manually as failing some thresholding criteria. This also is obviously cumbersome, and only works for `FMT` fields that are included in `stats` (e.g., not `GQ` or `PL`).
  
  I am not sure if I am missing some basic functionality of `bcftools`, or if this is more of a feature request. Any information would be appreciated.
  
  Thanks!
  
  **Tags:** vcf, snp, filtering, bcftools
  
  > **Comments:**
  > **username_1**: You may want to take a look at vcftools' genotype filtering options or the custom JEXL expressions that GATK's SelectVariants allows. Are you sure your sample-filtering operations won't produce a variable number of columns for each row? Also, remember that `bcftools filter` will not remove samples no matter what you try. It's a utility to filter sites, and it can only filter sites.  
  > **username_0**: Thanks very much.  Since every site has genotypes for every sample (if you include missing genotype entries), I'm not sure I understand how removing certain columns in their entirety would create any problems in terms of uneven dimensions.  Indeed, this is the function of `bcftools view -s ^<list.of.sample.IDs>`, if I am not mistaken.  `vcftools` also only allows for filtering samples based on ID.  Thank you for the reference to SelectVariants.  Do you have a link to a manual, or example code to address any of the example use-cases referenced in my question?  Thanks!  
  > **username_1**: Not really, no. I have rarely ever used SelectVariants when it wasn't already implemented in a pipeline (GATK tools can be a pain to use at times). If push comes to shove, it might be worth converting the VCF to MAF format so the data is at individual level.  
  
  ---
  
  ## 0 answers
  
  