Ok, so the problem must be on my setup. Did you run with pin_memory=True? Additionally, I was using torch.nn.DataParallel to train with 4 GPUs, and the dataset was stored in  torch.Tensors with the images already pre-processed, to avoid having to do the pre-processing during training. Do you think any of these factors may have caused the differences in CPU usage?
Thank you very much for your help!