I have a python script which gathers 10,000's of 'people' from an API and then goes on to request two other APIs to gather further data about them and then save the information to a local database, it takes around 0.9 seconds per person.
So at the moment it will take a very long time to complete. Would multi-threading help to speed this up? I tried a multi-threading test locally and it was slower, but this test was just a simple function without any API interaction or anything web/disk related.
thanks
How many cores do you have?
How parallelizable is the process?
Is the problem CPU bound?
If you have several cores and it's parallelizable across them, you're likely to get a speed boost. The overhead for multithreading isn't nearly 100% unless implemented awfully, so that's a plus.
On the other hand, if the slow part is CPU bound it might be a lot more fruitful to look into a C extension or Cython. Both of those at times can give a 100× speedup (sometimes more, often less, depending on how numeric the code is) for much less effort than a 2× speed-up with naïve usage of multiprocessing. Obviously the 100× speedup is only for the translated code.
But, seriously, profile. Chances are there are low hanging fruit that are much easier to access than any of this. Try a line profiler (say, the one called line_profiler [also called kernprof]) and the builtin cProfile.
Should i use multi-threading? (retrieving mass data from APIs)