I used the command watch -n0.1 nvidia-smi to check the behavior of GPUs and
I found the utilization of GPU becomes 0% at the beginning of each epoch for a short period. I am just wondering if it is common that the training hangs for a second before the beginning of each epoch? Maybe the reason is that dataloader has to re-prepare data for the beginning of each epoch?