Hi Guys,
When I run my AI train in multi-GPUs mode, process always stuck (hang) in backward. I traced it into torch interface torch/autograd/init.py line 145 Variable._execution_engine.run_backward. It was blocked process and never return. No matter what I have set up DataParallel or DistributedDataParallel.  The console nothing output. I guess something dead-loop in torch/_C.cpython-38-x86_64-linux-gnu.so or further. Have anyone know about this issue or workaround solution?
I compare two mode train (the single GPU and both GPUs), my train is successfully passed when only one GPUs but failed in double. And I collect the arguments of both of them when pass it into Variable.execution_engine.run_backward, they are same.
rt_frames.T = {NameError}name ‘rt_frames’ is not defined
create_graph = {bool} False
grad_tensors = {NoneType} None
grad_tensors = {tuple: 1} tensor(1., device=‘cuda:0’)
grad_variables = {NoneType} None
inputs = {tuple: 0} ()
retain_graph = {bool} False
tensors = {tuple: 1} tensor(0.6899, device=‘cuda:0’, grad_fn=)
And I also test GPU + CUDA with p2pBandwithLatencyTest and the result is OK.
Would anyone do me a favor on having a look at this issue?
My hardware environment is CPU Intel® Core™ i9-7900X CPU @ 3.30GHz, Memory 8G*8 DDR3,
GPU rtx3090
software environment is Ubuntu18.04TLS, RTX3090 Driver460.56 + cuda_11.1.TC455_06.29069683_0 + Tensorflow 2.4.1.+ Pytorch1.8.1(cu111).