Actually, even when i simplified my code down to the basics. just launching two processes for two GPU’s, as soon as the first process, hits the barrier, it stops the other process.
But, if i remove the loss.backward() and the optimizer stuff, it works