What is the best approach to scrape a big website?
2000 threads is too many.  Even 1 may be too many.  Your scraper will probably be thought of as a DOS (Denial Of Service) attach and your IP address will be blocked.
Even if you are allowed in, 2000 is too many threads.  You will bottleneck somewhere, and that chokepoint will probably lead to going slower than you could if you had some sane threading.  Suggest trying 10.  One way to look at it -- Each thread will flip-flop between fetching a URL (network intensive) and processing it (cpu intensive).  So, 2 times the number of CPUs is another likely limit.
You need a database under the covers.  This will let you top and restart the process.  More importantly, it will let you fix bugs and release a new crawler without necessarily throwing away all the scraped info.
The database will not be the slow part.  The main steps:

Pick a page to go for (and lock it in the database to avoid redundancy).
Fetch the page (this is perhaps the slowest part)
Parse the page (or this could be the slowest)
Store the results in the database
Repeat until no further pages -- which may be never, since the pages will be changing out from under you.

(I did this many years ago.  I had a tiny 0.5GB machine.  I quit after about a million analyzed pages.  There were still about a million pages waiting to be scanned.  And, yes, I was accused of a DOS attack.)
Hello I am developing a web scraper and I am using in a particular website, this website has a lot of URLs, maybe more than 1.000.000, and for scraping and getting the information I have the following architecture.
One set to store the visited sites and another set to store the non-visited sites.
For scraping the website I am using multithreading with a limit of 2000 threads.
This architecture has a problem with a memory size and can never finish because the program exceeds the memory with the URLs
Before putting a URL in the set of non-visited, I check first if this site is in visited, if the site was visited then I will never store in the non-visited sites.
For doing this I am using python, I think that maybe a better approach would be storing all sites in a database, but I fear that this can be slow
I can fix part of the problem by storing the set of visited URLs in a database like SQLite, but the problem is that the set of the non-visited URL is too big and exceeds all memory
Any idea about how to improve this, with another tool, language, architecture, etc...?
Thanks