How handle multithreading on big-scale web scraping project?
Ok, sounds smart :) Now I have 2 scripts with 'ThreadPool' method. Works fine. But to execute these scripts I have 'main_script' which until today I have separate threads for each script. So in main_script I can use too 'ThreadPool' method? And then how looks like a cost of CPU and generally perfomance? Because in future I want have e.g. 100 scripts in main_script (e.g. max_workers = 5) and every script have too ThreadPool (e.g. max_workers=5), so in one time I have a 5x5 = 25 threads?
I have a problem with my project. It involves downloading data from various websites.
For now it's tens of pages, but in the future it will be thousands of pages depending on the country. Each page has its own script.
To support these scripts, I created a main script that calls each subsequent script on a separate thread using the multi-threading library. 
The script on the input has a path to the file with data already downloaded to the database, and on the output it gives the second file only with new data. At the moment, this solution was used only for 4 pages. It works as I assume. But I wonder what happens when these websites will be a few hundred or a few thousand?
I think I could create a separate main script depending on the country, but as a result, it would give at least several hundred pages (scripts) per country, so according to my logic, several hundred threads run at one time.
Does it have a chance to operate on such a number of websites? I opted for multithreading due to the large number of web content download operations.