  The following paper presents various methods and implementation techniques
used to harvest metadata efficiently from a Kademlia Distributed Hashtable
(DHT) as used in the BitTorrent P2P network to build an index of publicly
available files in the BitTorrent ecosystem. The indexer design makes various
tradeoffs between throughput and fairness towards other DHT nodes while also
being scaleable in a distributed environment.

Networking and Internet Architecture