Modifying the number of workers didn’t change much, I tried 2,4 and 8 workers.
The collate_fn function is this one:

def collate_fn(batch):
    return tuple(zip(*batch))


Since my last comment, I’m using the float32 dtype and I’ve removed the torch.cuda.empty_cache() call. I’ve managed my meory a bit better with the del variable instructions given that I was also running out of cuda memory.
But only replacing the dataloader with a simple custom generator brought my performance from 3 seconds per image to 2,5 second per batch of 32 images.
I suspect it was doing something on the images themselves but I don’t know what.
Maybe there was something wrong with my dataset implementation that was slowing down the dataloader?
I’ve no clue but I’m also new to the whole PyTorch thing.