I want to crawl a webpage(news) and get only the latest links. 
I have a crawler code which gets all the links from a website and takes like 2-3 hours to get around 30000 links and stores in db.
What if next time when I run the crawler I want only the new links to be inserted.
I know I can do a filter before inserting in db, but I would want the crawler to fetch only new links other than crawling the old links again(basically entire website). 
Is it possible to do something like that?
You need some sort of a cache.
One solution which comes to my mind is storing a local version of the website. When you want to add the new links you can make a diff between the new version and your locally stored version. Afterwards you can crawl over the diff...
Is there a way to crawl a webpage in python where the crawler shoud fetch only the new links.