MOMSpider was the one I remember.<p>What&#x27;s interesting about web crawling is that it has always been controversial.<p>Many website operators back then did not like crawlers.  And that sort of sentiment continues to this day.  The web has never fully consented to being indexed.<p>This has always struck me as odd.  Perhaps there should have been more thought about a bifurcation of &quot;private&quot; web and &quot;public&quot; web.  Of course in retrospect the value of the public one seems far greater; Google&#x27;s corporate wealth points to that conclusion.