If I understand the issue correctly, your memory usage is increasing in each iteration.
This might happen if you are storing tensors, which are still attached to the computation graph, in e.g. a list.
Often you would like to append the loss to a list in order to calculate the mean for the epoch.
Since the loss tensor is attached to the computation graph, you would also store the complete graph in each iteration, which might eventually yield the OOM issue.
To detach the tensor properly, you could use:
losses.append(loss.cpu().detach().item())

The memory increase should not come from the options you’ve posted.