Suppose you want to call some API (open without any KEY needed) which has some rate limit per second (10 requests max per second). Suppose now, I am calling a function (with multiprocessing.Pool) which request data from API and process it. Is there some way to switch ip in order to not get blocked? Maybe a list of ip/ proxies. Can somebody tell me a way to get this done?
Thanks!
You unfortunately can't retrieve data from an api and do multiprocessing with different ip's because the api will already have your Ip assigned to its request variable. Some Web pages also have http 402 errors which means it is possible to get timed out for sending too many requests.
Python Multiprocessing Rate Limit