EvanZ:

According to some other help I saw online I added torch.cuda.empty_cache() but that is not fixing it.


Calling empty_cache() could potentially decrease the performance, since the cached memory will be freed and is thus not reusable. PyTorch would need to use synchronizing cudaMalloc operations in order to allocate new memory, which is the reason for the potential slowdown.
If you are not using other processes, which need to use GPU memory, I would not recommend to use empty_cache in your code.
That being said, if you are seeing a significant slowdown, you might (accidentally) store some tensors, which are still attached to the computation graph or your systems might be throttled e.g. because the cooling isn’t sufficient for the current workload (you could check the clocks of your system).