The memory leak is most likely not related to retain_graph but other changes you make. The only different retain_graph makes is that it delays the deletion of some buffers until the graph is deleted.
So the only way to these to leak is if you never delete the graph. But if you never delete it, even without retain_graph, you would end up running out of memory.
I would check using a package like torchviz 7 to make sure the graph does not grow forever. If it does, then you need to fix your code to avoid that (most likely missing .detach()).