I am unsure what tags to give this but I am using Selenium in python so I decided to start here. I am scraping a website thousands of times using selenium and requests in python. It starts fairly quickly but around the 3400 page load mark it slows down from around .1 seconds to 3 or 4 seconds. Any ideas on what is slowing the webpages loading. The program is being run on a very low power Linode (1 shared cpu and 1gb of ram). The cpu is pegged from the beginning when it is still running fast and from what I can tell, it is not using all the RAM. I also gave it a 10 gb swap. My internet download and upload is above 200 MB/s. I was thinking the website host themselves are limiting me but I don't know this stuff well enough to be sure.
Pretty sure it's the host. If they are limiting by your IP, you may want to use some proxies. If the website is on shared hosting or some low cost hosting then proxies won't help.
Websites are taking a long time to load when scraping