A common issue is storing the whole computation graph in each iteration.
Make sure you are not storing any tensors in e.g. a list or any other container, which might be still attached to the computation graph, as this will increase the memory usage in each iteration.
E.g. if you want to store the loss, use losses.append(loss.item()) instead of directly appending the loss.