I have substantial PHP experience, although I realize that PHP probably isn't the best language for a large-scale web crawler because a process can't run indefinitely. What languages do people suggest?
Any language you can easily use with a good network library and support for parsing the formats you want to crawl. Those are really the only qualifications.
What languages are good for writing a web crawler?