I guess there are three possible ways of answering this:


is it possible to avoid initialization? Possibly out of my own expertise to give a definitive answer. Simon Wang states that it’s not. That correlates with my own experience.


is it theoretically possible to reduce the footprint of initializing CUDA; and is it practically possible/realistic for pytorch to do so? For the first point, possibly out of my own experience; you could compile and run a simple C++ CUDA app that simply initializes cuda. There’s one at https://github.com/hughperkins/coriander/blob/master/test/endtoend/cuda_sample.cu 23 , which you can probably compile using nvcc or similar.  I guess that the intrinsic overhead is low. Is it practically possible/realistic for pytorch to do so. Definitely outside of my expertise. You’d need to make a compelling case that you’d exhausted all available options I would imagine, and that other people might encounter a similar issue


As far as ‘exhausting all available options’, I guess this is the bit I’m addressing mostly   Even if you somehow initialize CUDA with minimal overhead from eg 64 different processes, if they’re all running against the same GPU, the GPU bit is basically going to be sequential. In that case, could it be possible to have a single central process, that communicates via shared memory etc with the actor processes, and which handles the gpu processing for those multiple actor processes? I imagine this would run at the same speed as your ideal easy-to-program scenario, whilst avoiding the initialization issue?