Could be that your data loading is the bottleneck. Try to use the dataloader with multiple workers so that the dataset doesn’t not get loaded into the main process and sits ready for the next forward pass through the GPU.

My model mostly contains Conv2d and BatchNorm layers.

Not sure if you are doing convolutions on the CPU? Actually, that would be very different/way more inefficient than on GPU because the convolution algorithms on CPU and GPU are implemented very differently.