Right. I really should have been better about data provenance, because I know better than that.<p>On my disk, I do indeed see `2007.csv`, `2008.csv`, ..., `2013.csv`. Combined, they are 4.1GB uncompressed. It appears to contain 100,488,054 lines (50,244,026 CSV records). Since I only cared about indexing the URLs, I extracted them from the CSV data (this command just pulls out the second column from the CSV file, which is also one of my creations: <a href="https:&#x2F;&#x2F;github.com&#x2F;BurntSushi&#x2F;xsv" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;BurntSushi&#x2F;xsv</a> --- but a simple Python script should do nicely as well).<p><pre><code>    xsv select 2 20*.csv &gt; urls-unsorted
</code></pre>
And then remove quotes (because xsv correctly encodes CSV data):<p><pre><code>    sed -i &#x27;s&#x2F;^&quot;\(.\+\)&quot;$&#x2F;\1&#x2F;g&#x27; urls-sorted
</code></pre>
Finally, sort and dedupe them:<p><pre><code>    LC_ALL=C sort -u urls-unsorted &gt; urls-sorted

</code></pre>
I just ran this procedure again, and urls-sorted is byte-for-byte equivalent to the data I used in the blog post.<p>Judging by the timestamps on the original CSV files, I did this around the beginning of August. In fact, I had completely forgotten that I had to do this at all! Sorry about that. When I get a chance, I&#x27;ll update the blog post.