Thanks for the info.
Does the memory increase for the full training loop?
If so, I would try to debug where it occurs and maybe you can narrow it down to a specific line, e.g. the criterion or backward pass.
Then maybe you could create a small executable code snippet and we could have a look.