Writing your own search engine is hard (2004)
Yeah, there are certainly more problems these days. For one, the size of the web is larger, more of it is spam causing issues with pure page rank to detect networks that heavily link to each other.<p>Important sites have a bunch of anti-crawling detection set up (especially news sites). It's even worse that the best user-generated content is behind walled gardens in facebook groups, slack channels, quora threads, etc...<p>The rest of the good sites are javascript-heavy and you often have to run chrome headless to render the page and find the content - but that is detectable so you end up renting IP's from mobile number farms or trying to build your own 4G network.<p>On the upside, <a href="https://commoncrawl.org/" rel="nofollow">https://commoncrawl.org/</a> now exists and makes the prototype crawling work much easier. It's not the full internet, but gives you plenty to work with and test against so you can skip to the part where you figure out if you can produce anything useful should you actually try to crawl the whole internet.