Are you seeing the memory increase while (or after) calling backward() and is it stable afterwards?
If so, then note that the gradients would need to be stored on the device and will also take memory.
If you are seeing an increased memory usage of 10GB in each iteration, you are most likely storing the computation graph accidentally by e.g. appending the loss (without detaching it) to a list etc.