I don’t know, how utils.collate_fn is defined, but you might try to profile it or reduce the number of workers.
That being said, the double call will slow down your training additionally, as FP64 is slower on the GPU than FP32. While you will get more floating point precision, this is usually not necessary in the Deep Learning domain (but you might have your reasons to do so).
Also torch.cuda.empty_cache() will not save any memory, but only slow down your code, as PyTorch would need to reallocate the device memory with synchronizing calls.