Yeah, UB is undefined behaviour = anything could happen now.
@kmichaelkills we do release GIL in C++ parts and memory (de)allocation isn’t blocking. You can think of the caching allocator like a self expanding memory pool.
Multi-threaded multi-GPU works, and is used to dispatch kernels by our DataParallel module and data_parallel function. Still, even though we carefully release the GIL as soon as we enter C++, the contention on the lock, that serializes execution, makes it nearly impossible to saturate 8 modern GPUs. That’s why we’ve started exploring CUDA multiprocessing.
Regarding cffi, we didn’t want to add more dependencies, and having more fine grained control over GIL and some other aspects makes it easier for us to develop the C code. I’d love to support PyPy, but there are very few people using it, so it’s not really a top priority for us.
@florin well, the frames are on the CPU already, right? Can’t you cat them there and ship the whole batch to CUDA before giving it to the network? Why do you need CUDA multiprocessing there?
I don’t really understand the thing about multi-GPU. You can switch between GPUs using torch.cuda.device, and that’s all you need to execute code on another device. Alternatively, if you want to have purely asynchronous training, you could start up some threads or processes, have them run on separate GPUs, and sync parameters periodically. This is a use case where CUDA multiprocessing can come in handy, because the shared memory never goes out of scope.