I'm writing a spider in Python to crawl a site. Trouble is, I need to examine about 2.5 million pages, so I could really use some help making it optimized for speed.
What I need to do is examine the pages for a certain number, and if it is found to record the link to the page. The spider is very simple, it just needs to sort through a lot of pages.
I'm completely new to Python, but have used Java and C++ before. I have yet to start coding it, so any recommendations on libraries or frameworks to include would be great. Any optimization tips are also greatly appreciated.
You waste a lot of time waiting for network requests when spidering, so you'll definitely want to make your requests in parallel.  I would probably save the result data to disk and then have a second process looping over the files searching for the term.  That phase could easily be distributed across multiple machines if you needed extra performance.
Writing a Faster Python Spider