I was trying to measure the training time with different batch size. It shows that backward() and step() increase with larger batch size (forward time also increases but that’s expected). The following times are all averaged over all iterations in the epoch. (I believe I measured time correctly with torch.cuda.synchronize()
batch size = 96
Train Epoch #1: 100%|██████████| 3336/3336 [19:59<00:00, 2.78it/s, loss=5.85, top1=5.28, top5=15, data_time=0.00287, forward_time=0.0669, backward_time=0.197, cuda_time=0.00804, step_time=0.0417
batch size = 64
Train Epoch #1: 100%|█████████▉| 4995/5004 [19:58<00:02, 3.90it/s, loss=6.16, top1=2.9, top5=9.46, data_time=0.00134, forward_time=0.0444, backward_time=0.135, cuda_time=0.00387, step_time=0.0225
batch size = 32
Train Epoch #1: 100%|██████████| 10008/10008 [26:11<00:00, 6.37it/s, loss=6.84, top1=0.266, top5=1.22, data_time=0.000503, forward_time=0.0267, backward_time=0.0792, cuda_time=0.00204, step_time=0.0193]
Thanks!