Do you get this memory leak only with torch.autograph.set_detect_anomaly(True) or also without?
How much memory is your model using after the first epoch and how fast does the memory usage increase?
Also, which PyTorch version are you using?