torch.cuda.empty_cache() will not avoid the out of memory issue, but might instead just slow down your code, as PyTorch would need to reallocate the device memory.
I guess your memory usage grows, since you are storing the computation graphs for all time steps in memory before calling backward and thus freeing the intermediates.
You could either lower the number of steps or alternatively call backward() after each (or a couple of iterations) and detach() the hidden state in the following operation.
Let me know, if this would work for you.