Hello,
Because of GPU usage is not high in model training, I use DLProf to profile my model. The result shows that all_gather take a lot of time:

image1920×1095 103 KB


image1920×567 79.5 KB

Through profiling result I found these all_gather are called by sycnbn(pytorch/_functions.py at v1.8.2 · pytorch/pytorch · GitHub 1).
I want to know why it took so much time and how to speed up the training.
By the way, because of the large number of model parameters, I set batch_size=1 and use 8 V100 in training.