empty_cache will force PyTorch to reallocate the memory, if necessary, and thus might slow down the code.
The large cache might be created during the training, as the forward pass will create intermediate activations, which could be needed for the gradient calculation during the backward pass.
E.g. a simple nn.Conv2d might create an activation, which uses a multiple of the memory footprint of the input tensor, if the number of out_channels is large.