Is there a way to crawl a webpage in python where the crawler shoud fetch only the new links.
You need some sort of a cache.
One solution which comes to my mind is storing a local version of the website. When you want to add the new links you can make a diff between the new version and your locally stored version. Afterwards you can crawl over the diff...
I want to crawl a webpage(news) and get only the latest links. 
I have a crawler code which gets all the links from a website and takes like 2-3 hours to get around 30000 links and stores in db.
What if next time when I run the crawler I want only the new links to be inserted.
I know I can do a filter before inserting in db, but I would want the crawler to fetch only new links other than crawling the old links again(basically entire website). 
Is it possible to do something like that?