How to crawl world website like what google do
I'm wondering how to crawl entire websites in the world, like what google do.

I want create a search engine.

I know that SearchBlox/Solr/Lucid/[google mini][1] can crawl website, are they able to crawl the entire world website ?

How to do that ? is there any algorithm or software can do that. because i don't know how get all website name.


  [1]: http://www.google.com/enterprise/search/mini.html


And what the hardware infrastructure will be. please explain
How to crawl world website like what google do
===

I'm wondering how to crawl entire websites in the world, like what google do.

I want create a search engine.

I know that SearchBlox/Solr/Lucid/[google mini][1] can crawl website, are they able to crawl the entire world website ?

How to do that ? is there any algorithm or software can do that. because i don't know how get all website name.


  [1]: http://www.google.com/enterprise/search/mini.html


And what the hardware infrastructure will be. please explain