Solved by ptrblck in post #2 
              
              
                Are you storing some tensors in e.g. a list without detaching them? 
Also, could you use: 
with torch.no_grad():
    data_grad = data.grad

instead of calling the .data attribute, as it might yield unwanted side effects.