Are you storing some tensors in e.g. a list without detaching them?
Also, could you use:
with torch.no_grad():
    data_grad = data.grad

instead of calling the .data attribute, as it might yield unwanted side effects.