I changed the enviroment from pytorch 1.6 to latest nightly pytorch and the old problem start to change to new one. I used the same code to run the experiments, but barrier doesn’t work at all. The output is like:
Training:   0%|                                                                                                                                                                                                    | 0/5647 [00:18<?, ?it/s]Average loss :  2.0772855
rank 0 go to validation
start to validate
Training:   0%|                                                                                                                                                                                                    | 0/5647 [00:20<?, ?it/s]Average loss :  2.2598593
rank 1 go to barrier
rank 1 go out of barrier
Training:   0%|                                                                                                                                                                                                    | 0/5647 [00:00<?, ?it/s]
rank 0 go to barrier
rank 0 go out of barrier
The checkpoint has been saved to /home/dsv/qida0163/Vision/data/embedding/frcnn_coco_person/checkpoints/checkpoints_20201011_dis_1.pkl
Epoch 2/13
Training:   0%|
Then freezing.
Which means the barrier did work since the rank 1 go out before the validation and started the new batch training before rank 1 finish the validation.
I also tried to remove the barrier before, but it just goes freezing in the end.
Just want to provide more information.