If you see an increase in memory usage, you are most likely right in maintaining some references to variables.
Could you check your code for storing tensors which are not detached from the computation graph, e.g. losses.append(loss)? This is often the cause of increasing memory usage, as loss still holds the whole computation graph, if you don’t call detach() on it.