I’m a bit confused by the output logs. How does Rank 1 log “process initialized” after logging “training process passed data download barrier starting epochs”? Also why are there 3 ranks with a world size of 2? Lastly, “inside train method” should be logged twice if you are spawning 2 processes.
I reproduced your script, and swapped out the get_dataloader() and model.gru() calls with a print statement that prints “Just before barrier”. I got the expected output:
spawning
inside train method
inside train method
Rank 1/2 process initialized.
Rank 2/2 process initialized.
Rank 1/2 Just before barrier
Rank 1/2 training process passed data download barrier.
Rank 2/2 training process passed data download barrier.

barrier() should indeed block until all the nodes reach the barrier. It would be worth debugging if there is something odd occuring in the get_dataloader() and model.gru() calls. For a lighter weight barrier implementation, you can try calling dist.allreduce on a very small (or empty) torch tensor and this should mimic the barrier behavior.