I saw a very strange behavior during training. The Training slows down after a few steps (generally after 50% steps in the first epoch). At this point the GPU utilization (of 6 or 7 out of 8 GPUs) goes to 100% from 90%. While the remaining 1 or 2 of the GPUs go down to 0% utilization. No matter what training configuration i set, i see the same behavior. In this way i have one epoch in approximately 4 hours, instead of less than 10 minutes.
Environment:
Python Version: 3.6.9
[GCC 8.4.0]
PyTorch Version: 1.8.1+cu102
Number of GPUs: 8
I have no idea where is the bottleneck? Does anyone face similar problem?