An easy way to concurrency and parallelism with Python stdlib
I recently have been doing--what should be--straightforward subprocess work in Python, and the experience is infuriatingly bad. There are so many options for launching subprocesses and communicating with them, and each one has different caveats and undocumented limitations, especially around edge cases like processes crashing, timing out, killing them, if they are stuck in native code outside of the VM, etc.<p>For example, some high-level options include Popen, multiprocessing.Process, multiprocessing.Pool, futures.ProcessPoolExecutor, and huge frameworks like Ray.<p>multiprocessing.Process includes some pickling magic and you can pick from multiprocessing.Pipe and multiprocessing.Queue, but you need to use either multiprocessing.connection.wait() or select.select() to read the process sentinel simultaneously in case the process crashes. Which one? Well connection.wait() will not be interrupted by an OS signal. It&#x27;s unclear why I would ever use connection.wait() then, is there some tradeoff I don&#x27;t know about?<p>For my use cases, process reuse would have been nice to be able to reuse network connections and such (useful even for a single process). Then you&#x27;re looking at either multiprocessing.Pool or futures.ProcessPoolExecutor. They&#x27;re very similar, except some bug fixes have gone into futures.ProcessPoolExecutor but not multiprocessing.Pool because...???  For example, if your subprocess exits uncleanly, multiprocessing.Pool will just hang, whereas futures.ProcessPoolExecutor will raise a BrokenProcessPool and the pool will refuse to do any more work (both of these are unreasonable behaviors IMO). Timing out and forcibly killing the subprocess is its own adventure for each of these too. I don&#x27;t care about a result anymore after some time period passes, and they may be stuck in C code so I just want to whack the process and move on, but that is not very trivial with these.<p>What a nightmarish mess! So much for &quot;There should be one--and preferably only one--obvious way to do it&quot;...my God.<p>(I probably got some details wrong in the above rant, because there are so many to keep track of...)<p>My learning: there is no &quot;easy way to [process] parallelism&quot; in Python. There are many different ways to do it, and you need to know all the nuances of each and how they address your requirements to know whether you can reuse existing high-level impls or you need to write your own low-level impl.