It is interesting that switching back to register_backward_hook solves the issue. The issue with the memory leak should be that you are calling .backward() with create_graph=True, which creates a reference cycle. This is documented here: Automatic differentiation package - torch.autograd — PyTorch 1.8.1 documentation 1. We plan to add a warning for this soon…