I have heard of Python's GIL problem, which states that there can only be one Python thread executing the Python bytecode at one time in a multicore machine. So a multithreaded Python program is not a good idea.
I am wondering if I can write a C extension that uses pthread to potentially boost the performance of my program. I'd like to create a thread in my C extension and make it runs in parallel with Python's main thread.
My assumption is that the Python main thread will do things more related to IO, while the pthread in my C extension will spend most of its time doing computation. The Python main thread communicates with the thread in the C extension using a queue (like a producer consumer model).
Is there any difference between multithread with Python and C extension?
Provided one thread runs CPU bound while the other runs IO bound, I don't see a problem.
The IO bound thread will call IO routines which usually release the GIL while doing their stuff, effectively allowing the other thread to run.
So give the "simple" solution a try and switch only if it really doesn't work the way you want it to.
Could a C extension for multithreaded Python boost performance?