I’m not familiar with all the architecture in pytorch, but I would like to share some thoughts if I may. Wouldnt multi-threading mult-gpu work if:

C++ part release GIL systematically
Memory (de)allocation isnt blocking (use memory pool like Tensorflow instead of CachedAllocation)

I guess moving all c++ part to cffi would solve 1)  and would make it even possible to use pypy (lua torch  used JIT but in python its suddenly a bad idea?)