So I am writing link fetchers to find new links on particular sites for a given group of 'starting links'.
Currently I am using Python/Beautiful Soup to accomplish this with decent success.
I have an input file [for each site] that I build the 'starting links' list from.
I use urllib2 to load the webpages and then beautiful soup to find the group of links I need to fetch and append them to a list. Some sites have the links split between a lot of different pages so I have to load them all to collect the links.
After it collects all the specified type of links from each 'starting link', I then have it compare this list with a 'previously collected' list that I load from file. I then return the difference to another list which is the 'new links' list as well as add these to the 'previously collected' link list.
My problem is performance. I am recollecting all of these previously seen links each time I rerun the program which means I am reloading a bunch of pages that I am not going to get any new links from. 
Generally the sites add new links on top of the others, so I am thinking my next move might be to compare the 'currently availiable' link with the 'previously collected' list and if there is not a  match, then collect the link until a match occurs, where it would then drop out for this given 'starting link' and move on to the next, potentially saving a lot of page loads for sites the break up their links.
Does this make sense to help speed up the fetching of new links which I will schedule to run every few days? 
The 'previously collected' list could have a couple hundred thousand links in it, so I was not sure how this would effect things to run this comparison over and over vs keeping the program dumb and always recollecting all availiable links.
Do you have a better solution all together?  Any input is much appreciated.
You should consider using hashes for comparing the previously collected list.  Instead of storing a list of links as strings, store a list of MD5 or SHA1 hashes for those links.  Comparing a hash to a list of hashes is much faster than comparing a string to a list of strings.  
Or if you maintain and persist an actual hash table of encountered links, then you wouldn't have to do any searching and comparing through a list, but would have constant time lookup to know if you've seen a link.  A full hash table will cost a lot of memory if your list is big though.
Python Link Fetcher Performance Issue