Body: <p>I recently tried to reproduce population structure with 1000 Genomes Project data (Phase 3, 2504 people).</p>
  
  <p>According to supplementary material,</p>
  
  <blockquote>
    <p>To reduce the dataset, we used VCFtools to only keep biallelic, non-singleton SNV sites that were a minimum of 2 KB apart from each other. 
    These variants were then merged and converted into PLINK format, giving us 1,286,213 throughout all the autosomes. 
    PLINK was then used to filter the variants further, keeping only variants with a minor allele frequency of at least 0.05, 
    and to convert the variants into binary PED format. After this filter 193,634 sites remained with data from 2,504 individuals in 26 populations.</p>
  </blockquote>
  
  <p>And from <a href="ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/supporting/admixture_files/" rel="nofollow noreferrer">ftp repository</a>, I found the exact command-line to do it.</p>
  
  <ol>
  <li>To get biallelic, non-singleton SNVs, all minimum of 2KB apart from each other</li>
  </ol>
  
  <blockquote>
  <pre><code>./vcftools --gzvcf
  ALL.chr1.phase3_shapeit2_mvncall_integrated_v4.20130502.genotypes.vcf.gz
  --thin 2000 --min-alleles 2 --max-alleles 2 --non-ref-ac 2 --plink --chr 1 --out ALL.chr1.phase3_shapeit2_filtered.20141217
  </code></pre>
  </blockquote>
  
  <ol start="2">
  <li>To get variants with minor allele frequency of at least 0.05</li>
  </ol>
  
  <blockquote>
  <pre><code>./plink --file ALL.wgs.phase3_shapeit2_filtered.20141217.ped --out
  ALL.wgs.phase3_shapeit2_filtered.20141217.maf0.05 --make-bed --maf 0.05
  </code></pre>
  </blockquote>
  
  <ol start="3">
  <li>To get population structure with ADMIXTURE program (v1.23)</li>
  </ol>
  
  <blockquote>
    <p><code>./admixture ALL.wgs.phase3_shapeit2_filtered.20141217.maf0.05.bed K</code></p>
  </blockquote>
  
  <p>So I applied the same command (although versions of programs were not exactly the same) to the phase 3 data, and got different results (top: from supplementary data, bottom: my reproduction)
  <a href="https://i.sstatic.net/vnwUj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vnwUj.png" alt="enter image description here"></a></p>
  
  <p>I basically did the same thing as in the supplementary document:</p>
  
  <ol>
  <li><p>(for each chromosome \$ch) <code>bcftools view -v snps -O v $gzvcf | vcftools --vcf - --thin 2000 --min-alleles 2 --max-alleles 2 --non-ref-ac 2 --plink --chr ${ch} --out 1KGP_v5a_20130502.chr${ch}</code> [bcftools-1.9, vcftools 0.1.15]</p></li>
  <li><p>(for each chromosome \$ch) <code>$plink --file 1KGP_v5a_20130502.chr${ch} --maf 0.05 --make-bed --out 1KGP_v5a_20130502.chr${ch}.maf0.05</code> [plink 1.9b]</p></li>
  <li><p>(merge)</p></li>
  <li><p><code>admixture 1KGP_v5a_20130502.ALL.maf0.05.bed 5</code> [ADMIXTURE 1.3]</p></li>
  </ol>
  
  <p>I would like to know if there's some step I've missed on the process. Otherwise, I cannot understand why the population structure look so different.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T07:30:44.817Z
Comments.Text: Welcome to the site. As you mention that you use different software version: What versions are you using? Could you try the same version to see if you are able to reproduce the same figure?
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T14:11:02.623Z
Comments.Text: Did you perform your steps on all the chromosome or just chromosome 1?  Also in step 1 why did you skip the bedtools command that was used in the original paper?
Comments.UserDisplayName: Bioathlete
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-13T17:33:34.790Z
Comments.Text: @Llopis : I've noted the software versions I used at the end of command lines. When started with the same .bed/.bim files from the [repository] (ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/supporting/admixture_files/), I was able to get the same output with ADMIXTURE 1.3 as with ADMIXTURE 1.23 (original paper). So, at least, ADMIXTURE version doesn't seem to matter.
Comments.UserDisplayName: In-Hee Lee
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-13T17:40:29.787Z
Comments.Text: @Bioathlete : I did for all autosomes. I followed the steps in the [file] (ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/supporting/admixture_files/README.admixture_20141217) & supplementary section 9.7. I thought those steps can be directly applied to the released VCFs. Can you point me the bedtools command you mentioned?
Comments.UserDisplayName: In-Hee Lee
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-13T17:44:58.310Z
Comments.Text: sorry I meant bcftools not bedtools.  `bcftools view -v snps -O v $gzvcf `
Comments.UserDisplayName: Bioathlete
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-13T20:43:13.107Z
Comments.Text: @Bioathlete : The commands at the bottom were mine. Those at the top of the figure are from the supplementary document. Sorry about the confusion.
Comments.UserDisplayName: In-Hee Lee
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-13T23:13:23.447Z
Comments.Text: Ok then the question is reversed but still valid.  1) Why did you use bcftools? 2) Is the image from the paper using the whole genome or just chromosome 1?
Comments.UserDisplayName: Bioathlete
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-13T23:51:17.347Z
Comments.Text: @Bioathlete : 1) I used bcftools to select SNPs only, because it is described as they used SNVs. But the 1000G phase 3 VCF contained indels & SVs. 2) Image in the paper was also whole genome. After all, paper used 193,634 sites (over whole autosome) as input to ADMIXTURE, while I used 199,657 sites.
Comments.UserDisplayName: In-Hee Lee
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-13T08:31:15.520Z
Comments.Text: Did you check how many iterations they ran ADMIXTURE for, and how many you did? It may have yet to converge.
Comments.UserDisplayName: user438383
Tags: <human-genome><phylogenetics><figure-reproduction>
Title: Reproducing population structure from 1000 Genomes Data
ThreadText: # Reproducing population structure from 1000 Genomes Data
  
  (Asked by: username_0 on 2018-09-12)
  
  I recently tried to reproduce population structure with 1000 Genomes Project data (Phase 3, 2504 people).
  
  According to supplementary material,
  
  > To reduce the dataset, we used VCFtools to only keep biallelic, non-singleton SNV sites that were a minimum of 2 KB apart from each other.
  > These variants were then merged and converted into PLINK format, giving us 1,286,213 throughout all the autosomes.
  > PLINK was then used to filter the variants further, keeping only variants with a minor allele frequency of at least 0.05,
  > and to convert the variants into binary PED format. After this filter 193,634 sites remained with data from 2,504 individuals in 26 populations.
  
  And from [ftp repository](ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/supporting/admixture_files/), I found the exact command-line to do it.
  
  1. To get biallelic, non-singleton SNVs, all minimum of 2KB apart from each other
  
  > ```
  > ./vcftools --gzvcf
  > ALL.chr1.phase3_shapeit2_mvncall_integrated_v4.20130502.genotypes.vcf.gz
  > --thin 2000 --min-alleles 2 --max-alleles 2 --non-ref-ac 2 --plink --chr 1 --out ALL.chr1.phase3_shapeit2_filtered.20141217
  >
  > ```
  
  2. To get variants with minor allele frequency of at least 0.05
  
  > ```
  > ./plink --file ALL.wgs.phase3_shapeit2_filtered.20141217.ped --out
  > ALL.wgs.phase3_shapeit2_filtered.20141217.maf0.05 --make-bed --maf 0.05
  >
  > ```
  
  3. To get population structure with ADMIXTURE program (v1.23)
  
  > `./admixture ALL.wgs.phase3_shapeit2_filtered.20141217.maf0.05.bed K`
  
  So I applied the same command (although versions of programs were not exactly the same) to the phase 3 data, and got different results (top: from supplementary data, bottom: my reproduction)
  [![enter image description here](https://i.sstatic.net/vnwUj.png)](https://i.sstatic.net/vnwUj.png)
  
  I basically did the same thing as in the supplementary document:
  
  1. (for each chromosome \$ch) `bcftools view -v snps -O v $gzvcf | vcftools --vcf - --thin 2000 --min-alleles 2 --max-alleles 2 --non-ref-ac 2 --plink --chr ${ch} --out 1KGP_v5a_20130502.chr${ch}` [bcftools-1.9, vcftools 0.1.15]
  2. (for each chromosome \$ch) `$plink --file 1KGP_v5a_20130502.chr${ch} --maf 0.05 --make-bed --out 1KGP_v5a_20130502.chr${ch}.maf0.05` [plink 1.9b]
  3. (merge)
  4. `admixture 1KGP_v5a_20130502.ALL.maf0.05.bed 5` [ADMIXTURE 1.3]
  
  I would like to know if there's some step I've missed on the process. Otherwise, I cannot understand why the population structure look so different.
  
  **Tags:** human-genome, phylogenetics, figure-reproduction
  
  > **Comments:**
  > **username_1**: Welcome to the site. As you mention that you use different software version: What versions are you using? Could you try the same version to see if you are able to reproduce the same figure?  
  > **username_2**: Did you perform your steps on all the chromosome or just chromosome 1?  Also in step 1 why did you skip the bedtools command that was used in the original paper?  
  > **username_0**: @Llopis : I've noted the software versions I used at the end of command lines. When started with the same .bed/.bim files from the [repository] (ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/supporting/admixture_files/), I was able to get the same output with ADMIXTURE 1.3 as with ADMIXTURE 1.23 (original paper). So, at least, ADMIXTURE version doesn't seem to matter.  
  > **username_0**: @username_2 : I did for all autosomes. I followed the steps in the [file] (ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/supporting/admixture_files/README.admixture_20141217) & supplementary section 9.7. I thought those steps can be directly applied to the released VCFs. Can you point me the bedtools command you mentioned?  
  > **username_2**: sorry I meant bcftools not bedtools.  `bcftools view -v snps -O v $gzvcf `  
  > **username_0**: @username_2 : The commands at the bottom were mine. Those at the top of the figure are from the supplementary document. Sorry about the confusion.  
  > **username_2**: Ok then the question is reversed but still valid.  1) Why did you use bcftools? 2) Is the image from the paper using the whole genome or just chromosome 1?  
  > **username_0**: @username_2 : 1) I used bcftools to select SNPs only, because it is described as they used SNVs. But the 1000G phase 3 VCF contained indels & SVs. 2) Image in the paper was also whole genome. After all, paper used 193,634 sites (over whole autosome) as input to ADMIXTURE, while I used 199,657 sites.  
  > **username_3**: Did you check how many iterations they ran ADMIXTURE for, and how many you did? It may have yet to converge.  
  
  ---
  
  ## 0 answers
  
  