I have substantial PHP experience, although I realize that PHP probably isn't the best language for a large-scale web crawler because a process can't run indefinitely. What languages do people suggest?
You could consider using a combination of python and PyGtkMozEmbed or PyWebKitGtk plus javascript to create your spider.
The spidering could be done in javascript after the page and all other scripts have loaded.
You'd have one of the few web spiders that supports javascript, and might pick up some hidden stuff the others don't see :)
What languages are good for writing a web crawler?