I have substantial PHP experience, although I realize that PHP probably isn't the best language for a large-scale web crawler because a process can't run indefinitely. What languages do people suggest?
C++ - if you know what you're doing. You will not need a web server and a web application, because a web crawler is just a client, after all.
What languages are good for writing a web crawler?