Oh, that stack trace is all what I could get, since it just hangs without error.
Well, I guess it’s a kind of synchronization issue.
I have four networks netA, netB, netC, netD, which were
netA = nn.DataParallel(netA).cuda()
netB = nn.DataParallel(netB).cuda()
netC = netC.cuda(0)
netD = netD.cuda(1)
(I have two GPU devices)
Flow is
i (input) -> netA ---> netB -> x (output #1)
                   +-> netC -> y (output #2)
                   +-> netD -> z (output #3)

If this is not helpful to guess the cause, I would like to simplify my codes to reproduce the issue with minimal data upload.