yes,
but this still does not answer the question to why c++ multi-threading seems to be turned off when using ddp + multigpus.
i dont not whether the threads of each process block each other or something else.
the rest of computation is done on gpu where all tensors live.
only compute_cpu() is done exclusively on cpu because it is simply a c++ cpu implementation. dont have yet a gpu implementation.