torch.cuda-synchronize() should be called before the timer.
Nevertheless, it seems your DataLoader or Dataset takes that much time to load and process each sample. You could try to speed it up, e.g. by moving the data from an HDD to an SSD if possible, or try to speed up the preprocessing code.
Also, I’m currently unsure, why you can’t use multiple workers. Did you code just hang if you tried it?