Sorry for the late reply. Thanks a lot for your answer! That totally makes sense.
Regarding the use of ctx.save_for_backward, according to this comment 1, memory leak doesn’t seem to occur starting from PyTorch 0.4. Is that true? I’ll use ctx.save_for_backward anyway because it seems to be a safe option, thanks for the tip!