Thank you for your reply Piotr.
Both datasets were on the same device indeed. I never tried to remove the training code and test the data loader alone, but here is what I did:

I added logs on every operation (data loading, training) with time and time elapsed.
I also used the profiler from torch.utils.bottleneck, which is a magnificent tool.
I made sure I was not storing the computation graph history (e.g. by storing the loss value).
I searched around this forum extensively.

This investigation more or less made me think that there is a problem with the device. Indeed, it turned out to be an HDD and not an SDD. The consensus to approach this is:

Increase the num_workers in the data loader.
Use an SSD.

Which I did, and it fixed the issue, in the sense that now the model trains quicker and proportionally to the data size.
I still don’t understand why this is a fix, and why the time gap was so disproportional when I was using the HDD. But at least I can continue with my experiments!
Cheers.