torch.cuda.empty_cache() will, as the name suggests, empty the reusable GPU memory cache. PyTorch uses a custom memory allocator, which reuses freed memory, to avoid expensive and synchronizing cudaMalloc calls.
Since you are freeing this cache, PyTorch needs to reallocate the memory for each new data, which will slow down your code.