I was actually able to speed up the dataloader. Using pin_memory=True helped a lot, and I figured out I was storing some tensors attached to the computation graph in my early stopping function, which made the process a bit slower over time.
Right now I am using batches of 64 samples, the time to grab a batch is around 0.4 s and the overheads for forward + backward pass are around 22s. I am using 440x400 images since the features are very small.
P.s.: Times were measured using time.time(), which is probably wrong for the dataloading, since I am using num_workers = 8.