I have heard of Python's GIL problem, which states that there can only be one Python thread executing the Python bytecode at one time in a multicore machine. So a multithreaded Python program is not a good idea.
I am wondering if I can write a C extension that uses pthread to potentially boost the performance of my program. I'd like to create a thread in my C extension and make it runs in parallel with Python's main thread.
My assumption is that the Python main thread will do things more related to IO, while the pthread in my C extension will spend most of its time doing computation. The Python main thread communicates with the thread in the C extension using a queue (like a producer consumer model).
Is there any difference between multithread with Python and C extension?
The Python interpreter is not aware of C launched threads in any way, so they can happily churn their own CPU time.
However I doubt this is a correct solution for your performance problems. First try using multiple processes with multiprocess module. If interprocess IO is too much after that you can result trickery like C threads. This would make your program an order of magnitude more complex so avoid it if possible.
Could a C extension for multithreaded Python boost performance?