Solved by tom in post #2 
              
              
                Using torch.autograd.grad (which is a better fit conceptually) 
    dx_pred, = torch.autograd.grad(ys_pred, xs, torch.ones_like(ys_pred), create_graph=True)

removes the memory leak. 
Note that torch.tensor is preferred over the constructors like FloatTensor and we generally avoid .data. You don’t n…