I cannot reproduce this issue locally using a nightly binary from ~3 weeks ago.
If I run the model for 50 iterations and print the allocated device memory via print(torch.cuda.memory_allocated()), it seems to move between ~3521MB and ~3523MB.
How large is the increase in memory in your use case and are you eventually running out of memory?