Body: <p>I have merged two <strong>huge</strong> <code>.gen</code> files with roughly 500k samples in total - around 3 TB. </p>
  
  <p>I wish to run an IBD (identity by descent) check to identify if there are identical participants in the two cohorts before doing any further analyses. </p>
  
  <p>In the past I have simply used the <code>--genome</code> argument in PLINK and it would work fine however, in this instance the file is so large that doing such would cause the following error to appear: <em>Error: Out of memory.  The --memory flag may be helpful.</em></p>
  
  <p>I then opted to use PLINK v2 as it has a <code>--parallel</code> argument that, apparently, can be used alongside <code>--make-king</code> and/or <code>--make-king-table</code> to conduct this IBD check. I have had some trouble understanding the documentation surrounding these aforementioned arguments and would like some clarification as to how to ideally use them. Please assume that I am running on a multi-node, 24 core cluster.</p>
  
  <p>Here is the current code I am running:</p>
  
  <pre><code>plink2 --gen testo.gen \
  --sample testo.sample \
  --make-king \
  --make-king-table \
  --parallel 1 24
  </code></pre>
  
  <p>And here is what the PLINK documentation says about the arguments for this command:</p>
  
  <p><code>--parallel &lt;1-based current job index&gt; &lt;total job pieces&gt;</code></p>
  
  <p>Now, I have failed to understand what the first argument does, even after playing around with the parameters a little and extensively searching the web for an answer. </p>
  
  <p>What would changing this "1-based current job index" do? What does it mean? And for "total job pieces", will the adjustment of this parameter alter how many cores the jobs are distributed to or is it just an identifier as to how many compartmentalized sets are being made?</p>
  
  <p>If not, is there any way to specify which nodes you would like to run a job on and specify the number of cores as well? If yes, could you please provide an example? </p>
  
  <p>Thanks.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-19T22:10:32.860Z
Comments.Text: Here is an extensive documentation : https://www.cog-genomics.org/plink/1.9/parallel
Comments.UserDisplayName: Paul Endymion
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-20T02:05:44.217Z
Comments.Text: Two other notes: if you’re just trying to detect duplicates, you can use —king-table-filter to greatly reduce the output file size (and you can also afford to select a small random sample of higher-MAF variants, instead of brute-forcing the entire dataset); and if you’re running on a single machine, PLINK 2.0 doesn’t need —parallel, unlike PLINK 1.9 it’ll automatically adapt to the memory limit.
Comments.UserDisplayName: Christopher Chang
Tags: <snp><plink><genotyping>
Title: Running IBD using PLINK --parallel (v2)
ThreadText: # Running IBD using PLINK --parallel (v2)
  
  (Asked by: username_0 on 2019-12-19)
  
  I have merged two **huge** `.gen` files with roughly 500k samples in total - around 3 TB.
  
  I wish to run an IBD (identity by descent) check to identify if there are identical participants in the two cohorts before doing any further analyses.
  
  In the past I have simply used the `--genome` argument in PLINK and it would work fine however, in this instance the file is so large that doing such would cause the following error to appear: *Error: Out of memory. The --memory flag may be helpful.*
  
  I then opted to use PLINK v2 as it has a `--parallel` argument that, apparently, can be used alongside `--make-king` and/or `--make-king-table` to conduct this IBD check. I have had some trouble understanding the documentation surrounding these aforementioned arguments and would like some clarification as to how to ideally use them. Please assume that I am running on a multi-node, 24 core cluster.
  
  Here is the current code I am running:
  
  ```
  plink2 --gen testo.gen \
  --sample testo.sample \
  --make-king \
  --make-king-table \
  --parallel 1 24
  
  ```
  
  And here is what the PLINK documentation says about the arguments for this command:
  
  `--parallel <1-based current job index> <total job pieces>`
  
  Now, I have failed to understand what the first argument does, even after playing around with the parameters a little and extensively searching the web for an answer.
  
  What would changing this "1-based current job index" do? What does it mean? And for "total job pieces", will the adjustment of this parameter alter how many cores the jobs are distributed to or is it just an identifier as to how many compartmentalized sets are being made?
  
  If not, is there any way to specify which nodes you would like to run a job on and specify the number of cores as well? If yes, could you please provide an example?
  
  Thanks.
  
  **Tags:** snp, plink, genotyping
  
  > **Comments:**
  > **username_1**: Here is an extensive documentation : https://www.cog-genomics.org/plink/1.9/parallel  
  > **username_2**: Two other notes: if you’re just trying to detect duplicates, you can use —king-table-filter to greatly reduce the output file size (and you can also afford to select a small random sample of higher-MAF variants, instead of brute-forcing the entire dataset); and if you’re running on a single machine, PLINK 2.0 doesn’t need —parallel, unlike PLINK 1.9 it’ll automatically adapt to the memory limit.  
  
  ---
  
  ## 0 answers
  
  