Answers.Body: <p>Apparently there is a bug in ADMIXTURE that prevents for converging when using at the same time the haploid mode and multithreading (flags --haploid and -j respectively). The problem is solved by avoiding using one of them.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-10T19:03:38.293Z
Answers.LastActivityDate: 2020-08-10T19:03:38.293Z
Answers.UserDisplayName: Iriel
Body: <p>I am trying to run the ADMIXTURE software for the first time to analyse the structure of an in-house dataset of samples comparing to 1000 genomes project ancestral populations. I am trying to determine ancestry just from the X chromosome of those individuals.</p>
  <p>To do that, I have downloaded the X chromosome file from 1000 genomes (ALL.chrX.phase3_shapeit2_mvncall_integrated_v1b.20130502.genotypes.vcf.gz). I have removed duplicated with bcftools and converted to plink binary with the following command in plink 1.9:</p>
  <pre><code>plink --bcf ALL.chrX.phase3_shapeit2_mvncall_integrated_v1b.20130502.genotypes.bcf --keep-allele-order --vcf-idspace-to _ --const-fid --allow-extra-chr 0 --split-x b37 no-fail --make-bed
  </code></pre>
  <p>Next, I modified the .bim files to change the polymorphisms ID so them where the same as their position, as a way to standarize their names. Then I filtered related individuals, also with plink and found the intersection of .bim files from 1000 genomes and my in-house dataset (a bed file with microarray genotyped variants only from X non PAR region), that was already filtered to contain only variants genotyped against the plus strand (without indels).</p>
  <p>Then I merged both datasets with the following plink command:</p>
  <pre><code>plink --bfile filename1 --bmerge filename2.bed filename2.bim filename2.fam --make-bed --out filemerged
  </code></pre>
  <p>That command generated a file with all the SNPS that would have more than 3 alleles in the merged file so I removed all them from both datasets and merged them again. On that merged file I pruned SNPs by linkage disequilibrium:</p>
  <pre><code>plink --bfile cromXmerged2 --indep-pairwise 50 10 0.1
  </code></pre>
  <p>, assign sample sex on the .fam file and filtered individual with missingnes higher than 10%:</p>
  <pre><code>plink --file mergedfile --mind 0.1 --make-bed --out
  </code></pre>
  <p>I also set all heterozigous calls as missing with --set-hh-missing plink command.</p>
  <p>With the final input I runned a supervised ADMIXTURE as following:</p>
  <pre><code>./admixture finalinput.bed 3 --haploid=&quot;male:23&quot; --supervised -j4
  </code></pre>
  <p>Before running of course I created the finalinput.pop file in open office calc, by creating a list with the assigned population for each sample, with my in-house dataset left blanc. I just used african, european and east asian 1000 genomes samples because my own dataset has brazilian admixed samples. The final dataset has approximately 1500 SNPs and about 1600 individuals. My computer set up is CPU: i3-2370M with two cores, 4 threads@2.4 GHz, RAM 8 GB@1333 MHz and an SSD 120 GB Sata 3.0. OS Ubuntu 20.04 64 Bits.</p>
  <p>The thing is that ADMIXTURE have been running for three days without showing any error or producing result yet. I would like to know if it is normal to this analysis to take so long or if it could be any problem with my dataset.
  I would appreciate any suggestion or advice!!!
  Thanks in advance,
  Iriel</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T18:16:23.210Z
Comments.Text: This should take a few minutes, not 3 days! Definitely something seems to have gone wrong here. Are you running it locally or on a remote server? ADMIXTURE produces an output as its running, so you should check that it's doing that
Comments.UserDisplayName: user438383
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T18:48:37.103Z
Comments.Text: I am running it locally. And it did not produced any output at the moment. I tried a unsupervised analysis before with K = 3 to 10 and it took a few days to produce Q and P outputs for the first 2 K values. But I canceled that run because supervised analysis would be better for my goals.
Comments.UserDisplayName: Iriel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T18:53:34.757Z
Comments.Text: Did it literally not produce any output at all? I have just had a quick check and run it, and it produces this almost immediately:
  
  ``Parallel execution requested.  Will use 8 threads.
  Random seed: 43
  Point estimation method: Block relaxation algorithm``
Comments.UserDisplayName: user438383
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T19:16:15.607Z
Comments.Text: Nothing, it keeps showing these lines:743116 (QN/Block)  Elapsed: 0.354 Loglikelihood: -1.1648e+06 (delta): -103.745
  743117 (QN/Block)  Elapsed: 0.331 Loglikelihood: -1.16479e+06 (delta): 6.94675
  743118 (QN/Block)  Elapsed: 0.327 Loglikelihood: -1.16472e+06 (delta): 67.0046
Comments.UserDisplayName: Iriel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T19:19:05.060Z
Comments.Text: I mean, It did not produce any file yet.
Comments.UserDisplayName: Iriel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T21:08:17.760Z
Comments.Text: Seems like it's having trouble converging. There is a post here (https://www.biostars.org/p/368113/) which suggests that using  haploid mode and multithreading might cause issues converging and that you should try running it without multithreading. Actually, your dataset is so small that I doubt you need to multithread anyway :)
Comments.UserDisplayName: user438383
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T21:24:47.990Z
Comments.Text: Thanks for the reply! I will try without multithreading then.
Comments.UserDisplayName: Iriel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T07:25:52.410Z
Comments.Text: no problem. I am interested to see if this solves the problem.
Comments.UserDisplayName: user438383
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T17:47:12.367Z
Comments.Text: Yes! I took just 6 seconds to run!
Comments.UserDisplayName: Iriel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T18:03:32.500Z
Comments.Text: Great. Should add the solution as a response to this and accept the answer so other people who had a similar problem can find the solution.
Comments.UserDisplayName: user438383
Tags: <microarray><bed><phylogenetics>
Title: Time for running ADMIXTURE analysis
ThreadText: # Time for running ADMIXTURE analysis
  
  (Asked by: username_0 on 2020-08-09)
  
  I am trying to run the ADMIXTURE software for the first time to analyse the structure of an in-house dataset of samples comparing to 1000 genomes project ancestral populations. I am trying to determine ancestry just from the X chromosome of those individuals.
  
  To do that, I have downloaded the X chromosome file from 1000 genomes (ALL.chrX.phase3\_shapeit2\_mvncall\_integrated\_v1b.20130502.genotypes.vcf.gz). I have removed duplicated with bcftools and converted to plink binary with the following command in plink 1.9:
  
  ```
  plink --bcf ALL.chrX.phase3_shapeit2_mvncall_integrated_v1b.20130502.genotypes.bcf --keep-allele-order --vcf-idspace-to _ --const-fid --allow-extra-chr 0 --split-x b37 no-fail --make-bed
  
  ```
  
  Next, I modified the .bim files to change the polymorphisms ID so them where the same as their position, as a way to standarize their names. Then I filtered related individuals, also with plink and found the intersection of .bim files from 1000 genomes and my in-house dataset (a bed file with microarray genotyped variants only from X non PAR region), that was already filtered to contain only variants genotyped against the plus strand (without indels).
  
  Then I merged both datasets with the following plink command:
  
  ```
  plink --bfile filename1 --bmerge filename2.bed filename2.bim filename2.fam --make-bed --out filemerged
  
  ```
  
  That command generated a file with all the SNPS that would have more than 3 alleles in the merged file so I removed all them from both datasets and merged them again. On that merged file I pruned SNPs by linkage disequilibrium:
  
  ```
  plink --bfile cromXmerged2 --indep-pairwise 50 10 0.1
  
  ```
  
  , assign sample sex on the .fam file and filtered individual with missingnes higher than 10%:
  
  ```
  plink --file mergedfile --mind 0.1 --make-bed --out
  
  ```
  
  I also set all heterozigous calls as missing with --set-hh-missing plink command.
  
  With the final input I runned a supervised ADMIXTURE as following:
  
  ```
  ./admixture finalinput.bed 3 --haploid="male:23" --supervised -j4
  
  ```
  
  Before running of course I created the finalinput.pop file in open office calc, by creating a list with the assigned population for each sample, with my in-house dataset left blanc. I just used african, european and east asian 1000 genomes samples because my own dataset has brazilian admixed samples. The final dataset has approximately 1500 SNPs and about 1600 individuals. My computer set up is CPU: i3-2370M with two cores, 4 threads@2.4 GHz, RAM 8 GB@1333 MHz and an SSD 120 GB Sata 3.0. OS Ubuntu 20.04 64 Bits.
  
  The thing is that ADMIXTURE have been running for three days without showing any error or producing result yet. I would like to know if it is normal to this analysis to take so long or if it could be any problem with my dataset.
  I would appreciate any suggestion or advice!!!
  Thanks in advance,
  username_0
  
  **Tags:** microarray, bed, phylogenetics
  
  > **Comments:**
  > **username_1**: This should take a few minutes, not 3 days! Definitely something seems to have gone wrong here. Are you running it locally or on a remote server? ADMIXTURE produces an output as its running, so you should check that it's doing that  
  > **username_0**: I am running it locally. And it did not produced any output at the moment. I tried a unsupervised analysis before with K = 3 to 10 and it took a few days to produce Q and P outputs for the first 2 K values. But I canceled that run because supervised analysis would be better for my goals.  
  > **username_1**: Did it literally not produce any output at all? I have just had a quick check and run it, and it produces this almost immediately:  ``Parallel execution requested.  Will use 8 threads. Random seed: 43 Point estimation method: Block relaxation algorithm``  
  > **username_0**: Nothing, it keeps showing these lines:743116 (QN/Block)  Elapsed: 0.354 Loglikelihood: -1.1648e+06 (delta): -103.745 743117 (QN/Block)  Elapsed: 0.331 Loglikelihood: -1.16479e+06 (delta): 6.94675 743118 (QN/Block)  Elapsed: 0.327 Loglikelihood: -1.16472e+06 (delta): 67.0046  
  > **username_0**: I mean, It did not produce any file yet.  
  > **username_1**: Seems like it's having trouble converging. There is a post here (https://www.biostars.org/p/368113/) which suggests that using  haploid mode and multithreading might cause issues converging and that you should try running it without multithreading. Actually, your dataset is so small that I doubt you need to multithread anyway :)  
  > **username_0**: Thanks for the reply! I will try without multithreading then.  
  > **username_1**: no problem. I am interested to see if this solves the problem.  
  > **username_0**: Yes! I took just 6 seconds to run!  
  > **username_1**: Great. Should add the solution as a response to this and accept the answer so other people who had a similar problem can find the solution.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-08-10. Score: 2
  
  Apparently there is a bug in ADMIXTURE that prevents for converging when using at the same time the haploid mode and multithreading (flags --haploid and -j respectively). The problem is solved by avoiding using one of them.
  
  
  ---
  
  