Thanks a lot, that worked!
I have one question though, I broadcast process with local rank 5’s seconds as a tensor, and thus all the processes have the same seconds after broadcasting.
So how does it work, is it possible that a faster process uses an old value of the variable that is broadcasted later by the src process? Or do all the processes wait until the value has been broadcasted by the src variable?
Current time on machine is : 3 2020-08-02_21:52:52
Current time on machine is : 4 2020-08-02_21:52:52
Current time on machine is : 2 2020-08-02_21:52:52
Current time on machine is : 5 2020-08-02_21:52:52
Current time on machine is : 7 2020-08-02_21:52:52
Current time on machine is : 6 2020-08-02_21:52:53
Current time on machine is : 1 2020-08-02_21:52:53
Current time on machine is : 0 2020-08-02_21:52:53

Before Broadcasting seconds: tensor([52], device='cuda:4')
Before Broadcasting seconds: tensor([52], device='cuda:3')
Before Broadcasting seconds: tensor([52], device='cuda:2')
Before Broadcasting seconds: tensor([52], device='cuda:7')
Before Broadcasting seconds: tensor([52], device='cuda:5')
Before Broadcasting seconds: tensor([53], device='cuda:1')
Before Broadcasting seconds: tensor([53], device='cuda:0')
Before Broadcasting seconds: tensor([53], device='cuda:6')

<broadcast using torch.distributed.broadcast(LongTensor(seconds), src=5)>

After Broadcasting seconds  tensor([52], device='cuda:6')
After Broadcasting seconds  tensor([52], device='cuda:1')
After Broadcasting seconds  tensor([52], device='cuda:7')
After Broadcasting seconds  tensor([52], device='cuda:2')
After Broadcasting seconds  tensor([52], device='cuda:0')
After Broadcasting seconds  tensor([52], device='cuda:5')
After Broadcasting seconds  tensor([52], device='cuda:4')
After Broadcasting seconds  tensor([52], device='cuda:3')