<p>Hi,</p>

<p>Am looking for a fast way to parse through 2 large lists with millions of elements and identify the ones shared by both the lists, exclusive to list1, exclusive to list2. For smaller lists, perl hashes or the venny tool are useful, but I needed to do the same with these huge lists.</p>

<p>FWIW, these lists are actually read-ID from NGS data. The reads were aligned using 2 different approaches, and I wish to investigate of the ~ 30 million reads mapped by both approaches, how many are common, how many and which ones are aligned using only one approach and not the other. I mention this in case there is something in <a href='https://code.google.com/p/bedtools/'>bedtools</a>, bamtools or the like that might be relevant here.</p>

<p>thanks!</p>

<p>EDIT: Thanks for all the responses. I got bogged by other things and never got the chance to check on these. Will work my way through and add further comments/notes..</p>

<p>Assuming you are referring to IDs and you have lists that are structured something like the following:</p>

<pre><code>$ cat list-1.txt
foo
bar
baz
zab
oof

$ cat list-2.txt
zab
baz
zit
</code></pre>

<p>You can first sort the two files:</p>

<pre><code>$ sort list-1.txt &gt; list-1.sorted.txt
$ sort list-2.txt &gt; list-2.sorted.txt

$ cat list-1.sorted.txt
bar
baz
foo
oof
zab

$ cat list-2.sorted.txt
baz
zab
zit
</code></pre>

<p>Now, you can use join to find the elements that are common to both sets:</p>

<pre><code>$ join -1 1 -2 1 list-1.sorted.txt list-2.sorted.txt
baz
zab
</code></pre>

<p>Use the <code>-v</code> option to find those elements that are exclusive to set 1:</p>

<pre><code>join -v 1 -1 1 -2 1 list-1.sorted.txt list-2.sorted.txt
bar
foo
oof
</code></pre>

<p>Exclusive to set 2:</p>

<pre><code>join -v 2 -1 1 -2 1 list-1.sorted.txt list-2.sorted.txt
zit
</code></pre>

<p>Or, you can use the <code>comm</code> command to do it all in one step. The first column are elements exclusive to set 1, the second column is elements that are exclusive to set 2, and the third column is elements that are common to the two sets.  <code>join</code> is more flexible, but in simple cases like this, <code>comm</code> can be quite useful.</p>

<pre><code>$ comm list-1.sorted.txt list-2.sorted.txt 
bar
        baz
foo
oof
        zab
    zit
</code></pre>
