Answers.Body: <p>First comment: you should be using real data, rather than simulated data, as we are not very good at modelling genetic variation and recombination at a genome-wide scale. A way that this can be done with GCTA is mentioned in the <a href="https://www.cog-genomics.org/plink/1.9/input#random" rel="nofollow noreferrer">PLINK manual</a>.</p>
  
  <p>I'll do some calculations to estimate the size of data you need to store:</p>
  
  <p>Assume that this plink-formatted file is in tped format (one row per SNP), and the entire genome is diploid. That means that a single SNP/individual pair will take up three bytes. You want 30,000 cases and 70,000 controls, so each line from each sample will take up 300kB (plus a bit of change for the header). With 1 thousand SNPs, that's 300MB per sample; with 1 million SNPs, that's 300GB per sample; with 10 million SNPs, that's 3TB per sample; with 40 million SNPs, that's 12 TB for one sample.</p>
  
  <p>Reading the plink manual, I see that the BED format is a two-bit compression for each SNP/individual pair, so one byte per 4 individuals, or about 12x compression (i.e. 1TB for one sample).</p>
  
  <p>Okay, that's a lot. It's well within a 256TB storage limit, but might hit file size limits depending on the file system.</p>
  
  <p>If there is a data generation issue with PLINK, it might be worth trying (as you suggest) splitting the input for simulation then merging afterwards. My guess is that doing the simulation on fewer SNPs (rather than fewer individuals) would work better. Plink has a <code>--bmerge</code> <a href="https://www.cog-genomics.org/plink/1.9/data#merge" rel="nofollow noreferrer">option</a> for merging the compressed binary file formats.</p>
  
  <p>I usually approach problems like these by working my way up from something that I know works. Try first merging two datasets of 1000 SNPs. If that works, try larger datasets. Rinse and repeat until the computer crashes, you get bored, or you reach your goal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-15T11:26:33.150Z
Answers.Text: Hi @gringer thanks for the answer! Do you know where can I find free and open license real data with this amount of samples? I have been using the 1000 genome phase 3, but they have only 2504 samples. For this one, I managed to create synthetic phenotypes through some scripts I put together (https://github.com/bambrozio/bioinformatics/blob/master/utils/1k-genomes-phenotype-simulator.ipynb), but  GCTA --simu-cc/--simu-qt probably would address it in a more reliable way. Thanks for that!
Answers.UserDisplayName: Bruno Ambrozio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-15T11:28:10.773Z
Answers.Text: For now, I managed to create a 20k samples with 8mi SNPs with Plink (1/5 from my final goal). My limitations is actually the S3 file size limitations. Also, your maths make sense, and suggests my scenarios doesn't. Thanks for that!
  Btw - loved your TEDxWellington talk :)
Answers.UserDisplayName: Bruno Ambrozio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-12T10:57:33.907Z
Answers.LastActivityDate: 2020-02-12T10:57:33.907Z
Answers.UserDisplayName: gringer
Body: <p><em>Note: this question can also be found on <a href="https://www.biostars.org/p/421485/" rel="nofollow noreferrer">Biostars</a></em></p>
  
  <p>I need to perform a stress test in a GWAS tool and the duty demands a dataset (plink format) having 100 thousand samples, having 40 million SNPs in a proportion of case:control=30:70. I'm performing the command:</p>
  
  <blockquote>
    <p>plink1.9 --simulate ds1.sim --make-bed --out ds1 --simulate-ncases 30000 --simulate-ncontrols 70000</p>
  </blockquote>
  
  <p>But after about 3 hours execution, I'm facing <code>Error: File write failure.</code>, even though my environment has 256T of disk. (It's a container with a S3 (Object store) mounted through s3fs).</p>
  
  <p>I was wondering if there's a clever way to do that? Perhaps breaking it into small chunks and merging up later somehow? It would prevent me to be waiting hours till an error happens and I loose either time and the sample generated so far.</p>
  
  <p>Thanks in advance!</p>
  
Tags: <gwas><plink>
Title: Question: How to simulate 100k samples having 40 million SNPs in a proportion of case:control=30:70?
ThreadText: # Question: How to simulate 100k samples having 40 million SNPs in a proportion of case:control=30:70?
  
  (Asked by: username_0 on 2020-02-11)
  
  *Note: this question can also be found on [Biostars](https://www.biostars.org/p/421485/)*
  
  I need to perform a stress test in a GWAS tool and the duty demands a dataset (plink format) having 100 thousand samples, having 40 million SNPs in a proportion of case:control=30:70. I'm performing the command:
  
  > plink1.9 --simulate ds1.sim --make-bed --out ds1 --simulate-ncases 30000 --simulate-ncontrols 70000
  
  But after about 3 hours execution, I'm facing `Error: File write failure.`, even though my environment has 256T of disk. (It's a container with a S3 (Object store) mounted through s3fs).
  
  I was wondering if there's a clever way to do that? Perhaps breaking it into small chunks and merging up later somehow? It would prevent me to be waiting hours till an error happens and I loose either time and the sample generated so far.
  
  Thanks in advance!
  
  **Tags:** gwas, plink
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-12. Score: 1 | Accepted answer
  
  First comment: you should be using real data, rather than simulated data, as we are not very good at modelling genetic variation and recombination at a genome-wide scale. A way that this can be done with GCTA is mentioned in the [PLINK manual](https://www.cog-genomics.org/plink/1.9/input#random).
  
  I'll do some calculations to estimate the size of data you need to store:
  
  Assume that this plink-formatted file is in tped format (one row per SNP), and the entire genome is diploid. That means that a single SNP/individual pair will take up three bytes. You want 30,000 cases and 70,000 controls, so each line from each sample will take up 300kB (plus a bit of change for the header). With 1 thousand SNPs, that's 300MB per sample; with 1 million SNPs, that's 300GB per sample; with 10 million SNPs, that's 3TB per sample; with 40 million SNPs, that's 12 TB for one sample.
  
  Reading the plink manual, I see that the BED format is a two-bit compression for each SNP/individual pair, so one byte per 4 individuals, or about 12x compression (i.e. 1TB for one sample).
  
  Okay, that's a lot. It's well within a 256TB storage limit, but might hit file size limits depending on the file system.
  
  If there is a data generation issue with PLINK, it might be worth trying (as you suggest) splitting the input for simulation then merging afterwards. My guess is that doing the simulation on fewer SNPs (rather than fewer individuals) would work better. Plink has a `--bmerge` [option](https://www.cog-genomics.org/plink/1.9/data#merge) for merging the compressed binary file formats.
  
  I usually approach problems like these by working my way up from something that I know works. Try first merging two datasets of 1000 SNPs. If that works, try larger datasets. Rinse and repeat until the computer crashes, you get bored, or you reach your goal.
  
  > **Comments:**
  > **username_0**: Hi @username_1 thanks for the answer! Do you know where can I find free and open license real data with this amount of samples? I have been using the 1000 genome phase 3, but they have only 2504 samples. For this one, I managed to create synthetic phenotypes through some scripts I put together (https://github.com/bambrozio/bioinformatics/blob/master/utils/1k-genomes-phenotype-simulator.ipynb), but  GCTA --simu-cc/--simu-qt probably would address it in a more reliable way. Thanks for that!  
  > **username_0**: For now, I managed to create a 20k samples with 8mi SNPs with Plink (1/5 from my final goal). My limitations is actually the S3 file size limitations. Also, your maths make sense, and suggests my scenarios doesn't. Thanks for that! Btw - loved your TEDxWellington talk :)  
  
  ---
  
  