Is it true that you can increase your batch size up till your ~maximum GPU memory before loss.step() slows down? I thought a GPU would do computation for all samples in the batch in parallel, but it seems like Pytorch GPU-accelerated backprop takes much longer for bigger batches. It could be swapping to CPU, but I look at nvidia-smi Volatile GPU Memory usage and it is under 70%.