I have a very large text file (9.2GB) that contains data in two columns (named All_SNPs.txt). I would like to use this file to convert my chr:pos to rs in my plink files. The file looks like this: 

    1:116342	rs1000277323
    1:173516	rs1000447106
    1:168592	rs1000479828
    1:102498	rs1000493007


However, `plink` produced an error stating that All_SNPs.txt contains duplicates in column 1. I  have tried different ways of removing the duplicates; specifically, I attempted the following line commands: 

    awk '!x[$1]++ { print $1,$2 }' All_SNPs.txt > All_SNPs_nodup.txt
    
    sort  All_SNPs.txt | uniq -u > All_SNPs_nodup.txt
    
    cat All_SNPs.txt | sort | uniq -u > All_SNPs_nodup.txt

However, each time I faced the same two problems: 1) either, the code will not make any difference and I will receive the same file back (this is for line commands that uses `cat`) or 2) I will receive an error : the procedures halted: have exceeded (this is for `sort | uniq`, and `awk`). 

I will be very grateful for any ideas of how I can make this work. Thank you very much.

PS, this file is far too large to open it in R. 
Split the text file into smaller one using the `split` command. You could split by size, as example:

    split -b 200m filename

This will produce files named ' xaa, xab, xac..' . Now use awk, but with simpler syntax

    awk -F"\t" '!seen[$1]++' xa*

And after that, join files using a sample `cat` into the destination file.