Answers.Body: <p>I contacted the QCTOOL team directly and received this response:</p>
  <blockquote>
  <p>The design of BGEN means that when you subset samples the data has to
  be recompressed - this is essentially what makes this slow.  (By
  contrast you can subset SNPs very quickly without recompression using
  bgenix <a href="https://code.enkre.net/bgen." rel="nofollow noreferrer">https://code.enkre.net/bgen.</a>)  It is therefore definitely worth
  considering not subsetting samples but using a sample
  inclusion/exclusion list instead, if your analysis software supports
  that.</p>
  <p>If you have to subset and want to use QCTOOL - some things to try are:
  I typically use the options -bgen-compression zstd -bgen-bits 8 now,
  c.f. <a href="https://doi.org/10.1101/308296" rel="nofollow noreferrer">https://doi.org/10.1101/308296</a> this is faster but first check
  your downstream software supports zstd compression. Use a map/reduce
  type pipeline (i.e. chunk data for re-encoding) - this can be
  implemented using bgenix and cat-bgen.</p>
  </blockquote>
  <blockquote>
  <p>I assume you’re working with the latest version?</p>
  <p>Have you tried first stripping the SNPs, then stripping the samples?</p>
  <p>Ever since working with the imputed data from the UKB I was never
  successful in chopping pieces of bgen files using qctool in a timely
  manner. If my memory serves me well, qctool manages to strip SNPs
  fairly quickly, but is super slow to remove samples. Try PLINK 1.9 or
  2.0.</p>
  <p>As for “The computational facility I'm using should not limit the speed of an operation.”,
  depending on how disk data moves in/out of slave nodes and how busy
  the cluster is, I did saw processes to become I/O starved on large
  clusters.</p>
  <p>Also, consider that most tools that perform association testing can
  utilize SNPs  and samples lists. BOLT, SAIGE, regenie, SNPtest… so
  there might not be a need to pre-filter. Put “NA” as phenotypes you’d
  like to be “removed” from the testing.</p>
  </blockquote>
  <p>In the end I did not pre-filter SNPs or samples - I set samples to NA within the phenotype file, and used a SNP inclusion list in the second stage of SAIGE with the flag <code>idstoIncludeFile</code>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-15T03:26:15.210Z
Answers.Text: When subsetting SNPs (and keeping all samples), I find bgenix to be about 100x faster than qctool.
Answers.UserDisplayName: carbocation
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-06T10:52:51.283Z
Answers.LastActivityDate: 2022-06-06T10:52:51.283Z
Answers.UserDisplayName: icedcoffee
Body: <p>I’m currently using QCTOOL v2 to process imputed .bgen files from UK Biobank, however they seem to be processing very slowly. Is this normal?</p>
  <p>My command is pretty basic; I’m filtering out a list of SNPs and samples:</p>
  <pre><code>/path_to/qctool \
  -g /path_to/ukbXXXX_c22_b0_v3.bgen \
  -s /path_to/ukbXXXX_c22_b0_v3.sample \
  -og /output_path/ukb_c22_filt.bgen \
  -os /output_path/ukb_c22_filt.sample \
  -excl-rsids /path_to/snps_rem_c22.txt \
  -excl-samples /path_to/samples_to_rem.txt
  </code></pre>
  <p>It is currently processing SNPs at a rate of <strong>1.2/s</strong> (e.g. 71169/?,57205.8s,1.2/s).</p>
  <p>The computational facility I'm using should not limit the speed of an operation.</p>
  <ul>
  <li>Have I made any mistakes in my qctool query?</li>
  <li>Or any other ideas on why it's running so slowly?</li>
  </ul>
  <p>Alternative suggestions on how to process these files would be appreciated for example, although I would prefer to use QCTOOL perhaps I have to use PLINK.</p>
  <hr />
  <p>From the website, <a href="https://www.well.ox.ac.uk/%7Egav/qctool_v2/" rel="nofollow noreferrer">https://www.well.ox.ac.uk/~gav/qctool_v2/</a></p>
  <p>&quot;QCTOOL is a command-line utility program for manipulation and quality control of gwas datasets and other genome-wide data&quot;</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-16T10:38:29.437Z
Comments.Text: Hi @Cat could you provide information on computer(s) you are using, e.g. are you using a desktop, laptop or the cloud? What's the RAM of the machine, how full is the hard drive and how big are the files, that sort of thing.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-16T19:14:47.867Z
Comments.Text: crossposted https://www.biostars.org/p/9519364/
Comments.UserDisplayName: user438383
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-18T12:55:22.113Z
Comments.Text: Also whether the files (either input files or output files) are on a local or network drive, that can make a huge difference to the speed.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-04T13:53:27.090Z
Comments.Text: bgen/qctool2 is just pretty slow in general I've found. For performance, convert to pgen format and use plink2, seems to be orders of magnitude faster as long as you have enough RAM to hold the files in memory,
Comments.UserDisplayName: user438383
Tags: <plink><qc><ld>
Title: Using QCTOOL v2 to process UK Biobank .bgen files - why so slow?
ThreadText: # Using QCTOOL v2 to process UK Biobank .bgen files - why so slow?
  
  (Asked by: username_0 on 2022-04-16)
  
  I’m currently using QCTOOL v2 to process imputed .bgen files from UK Biobank, however they seem to be processing very slowly. Is this normal?
  
  My command is pretty basic; I’m filtering out a list of SNPs and samples:
  
  ```
  /path_to/qctool \
  -g /path_to/ukbXXXX_c22_b0_v3.bgen \
  -s /path_to/ukbXXXX_c22_b0_v3.sample \
  -og /output_path/ukb_c22_filt.bgen \
  -os /output_path/ukb_c22_filt.sample \
  -excl-rsids /path_to/snps_rem_c22.txt \
  -excl-samples /path_to/samples_to_rem.txt
  
  ```
  
  It is currently processing SNPs at a rate of **1.2/s** (e.g. 71169/?,57205.8s,1.2/s).
  
  The computational facility I'm using should not limit the speed of an operation.
  
  * Have I made any mistakes in my qctool query?
  * Or any other ideas on why it's running so slowly?
  
  Alternative suggestions on how to process these files would be appreciated for example, although I would prefer to use QCTOOL perhaps I have to use PLINK.
  
  ---
  
  From the website, [https://www.well.ox.ac.uk/~gav/qctool\_v2/](https://www.well.ox.ac.uk/%7Egav/qctool_v2/)
  
  "QCTOOL is a command-line utility program for manipulation and quality control of gwas datasets and other genome-wide data"
  
  **Tags:** plink, qc, ld
  
  > **Comments:**
  > **username_1**: Hi @username_0 could you provide information on computer(s) you are using, e.g. are you using a desktop, laptop or the cloud? What's the RAM of the machine, how full is the hard drive and how big are the files, that sort of thing.  
  > **username_2**: crossposted https://www.biostars.org/p/9519364/  
  > **username_3**: Also whether the files (either input files or output files) are on a local or network drive, that can make a huge difference to the speed.  
  > **username_2**: bgen/qctool2 is just pretty slow in general I've found. For performance, convert to pgen format and use plink2, seems to be orders of magnitude faster as long as you have enough RAM to hold the files in memory,  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2022-06-06. Score: 1
  
  I contacted the QCTOOL team directly and received this response:
  
  > The design of BGEN means that when you subset samples the data has to
  > be recompressed - this is essentially what makes this slow. (By
  > contrast you can subset SNPs very quickly without recompression using
  > bgenix <https://code.enkre.net/bgen.>) It is therefore definitely worth
  > considering not subsetting samples but using a sample
  > inclusion/exclusion list instead, if your analysis software supports
  > that.
  >
  > If you have to subset and want to use QCTOOL - some things to try are:
  > I typically use the options -bgen-compression zstd -bgen-bits 8 now,
  > c.f. <https://doi.org/10.1101/308296> this is faster but first check
  > your downstream software supports zstd compression. Use a map/reduce
  > type pipeline (i.e. chunk data for re-encoding) - this can be
  > implemented using bgenix and cat-bgen.
  
  > I assume you’re working with the latest version?
  >
  > Have you tried first stripping the SNPs, then stripping the samples?
  >
  > Ever since working with the imputed data from the UKB I was never
  > successful in chopping pieces of bgen files using qctool in a timely
  > manner. If my memory serves me well, qctool manages to strip SNPs
  > fairly quickly, but is super slow to remove samples. Try PLINK 1.9 or
  > 2.0.
  >
  > As for “The computational facility I'm using should not limit the speed of an operation.”,
  > depending on how disk data moves in/out of slave nodes and how busy
  > the cluster is, I did saw processes to become I/O starved on large
  > clusters.
  >
  > Also, consider that most tools that perform association testing can
  > utilize SNPs and samples lists. BOLT, SAIGE, regenie, SNPtest… so
  > there might not be a need to pre-filter. Put “NA” as phenotypes you’d
  > like to be “removed” from the testing.
  
  In the end I did not pre-filter SNPs or samples - I set samples to NA within the phenotype file, and used a SNP inclusion list in the second stage of SAIGE with the flag `idstoIncludeFile`.
  
  > **Comments:**
  > **username_5**: When subsetting SNPs (and keeping all samples), I find bgenix to be about 100x faster than qctool.  
  
  ---
  
  