Solved by zfzhang in post #5 
              
              
                I came across torch.utils.checkpoint, wondering if it would solve the problem. 

Edit: I confirm that checkpoint solves the problem, at the expense of more computational time.