Thank you very much @mrshenli. I believe it got halted in the second torch.distributed.barrier. Because I could see the data download/preprocessing was successful on both nodes:
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified

I was using the following command to start this time, only changing the number of gpus from 8 to 4 for each node. It runs fine with or without barriers if I only train using one single node.
$ python -m torch.distributed.launch --nproc_per_node=4 --nnodes=2 --node_rank=0 --master_addr="192.168.0.1" --master_port=1234 resnet_ddp.py
$ python -m torch.distributed.launch --nproc_per_node=4 --nnodes=2 --node_rank=1 --master_addr="192.168.0.1" --master_port=1234 resnet_ddp.py

I have not tried removing the saving model code but will give it a shot.