I have a very large text file (9.2GB) that contains data in two columns (named All_SNPs.txt). I would like to use this file to convert my chr:pos to rs in my plink files. The file looks like this: 

    1:116342	rs1000277323
    1:173516	rs1000447106
    1:168592	rs1000479828
    1:102498	rs1000493007


However, `plink` produced an error stating that All_SNPs.txt contains duplicates in column 1. I  have tried different ways of removing the duplicates; specifically, I attempted the following line commands: 

    awk '!x[$1]++ { print $1,$2 }' All_SNPs.txt > All_SNPs_nodup.txt
    
    sort  All_SNPs.txt | uniq -u > All_SNPs_nodup.txt
    
    cat All_SNPs.txt | sort | uniq -u > All_SNPs_nodup.txt

However, each time I faced the same two problems: 1) either, the code will not make any difference and I will receive the same file back (this is for line commands that uses `cat`) or 2) I will receive an error : the procedures halted: have exceeded (this is for `sort | uniq`, and `awk`). 

I will be very grateful for any ideas of how I can make this work. Thank you very much.

PS, this file is far too large to open it in R. 
Cut and uniq to find duplicates, then grep -v them away. Relatively quick on 389MB dummy file.

    time sort -V All_SNPs.txt | cut -f 1 | uniq -c | perl -ane 'if($F[0] ne "1"){print "$F[1]\t$F[0]\n";'} > All_SNPs.dup.chr-pos.txt

    real    0m57.428s
    user    3m19.091s
    sys     0m3.152s
        
    time cut -f 1 All_SNPs.dup.chr-pos.txt | grep -wvf - All_SNPs.txt  > All_SNPs.nodup.txt

    real    0m2.516s
    user    0m1.697s
    sys     0m0.229s


