Hi!
I’m trying to learn a differential equation from data, for that I defined a net that approximates the derivative at a point. I then use the output of the net to approximate the next grid-point of the solution to the differential equation and use that as an input for the next step. The function calls of the net are therefore recursive. Now I also know already pretty exactly the formula of the differential equation i want to approximate and do some feature extraction in every step, precisely building some products, powers, squares, etc with the output of the net and additional parameters.
class ODEMultistep(nn.Module):

    def __init__(self, args): 
        super(ODEMultistep, self).__init__()
        self.fnn1 = Derivative_FNN().to(args.device)
        self.fnn2 = Superpos_FNN(args.steps).to(args.device) 
        self.args = args

    def forward(self, nr, iv, k, stop):
        ni_pred = torch.zeros(nr.size(0), stop).to(self.args.device)
        ni_pred[:, :self.args.steps] = iv.clone().detach()
        ni_pred.requires_grad = True

        # this loops over all available nodes and calculates the next node
        for j in range(self.args.steps, stop - self.args.steps):

            solutions = []
            # this calculates all previous derivatives
            for l in range(j - self.args.steps, j):
                t1 = nr[:, l] / ni_pred[:, l] * nr[:, l] # here calculate features
                td = nr[:, l].pow(2)
                td -= ni_pred[:, l].pow(2) # in this line the gradient gets lost
                t2 = k * nr[:, l]
                t2 *= torch.sqrt(td)
                t1, t2 = t1.view(-1, 1), t2.view(-1, 1)
                derivative = self.fnn1(t1, t2)
                solutions.append(derivative)
            this_derivative = self.fnn2(solutions)
            ni_pred[:, j] = ni_pred[:, j - 1] + self.args.stepsize * this_derivative

        return ni_pred

Now I get this error message when running the code:
sys:1: RuntimeWarning: Traceback of forward call that caused the error:
  File "/home/felixwagner/MEGAsync/Projektarbeit/Code/train_ODEMultistep.py", line 164, in <module>
    train(args, model, criterion, train_loader, optimizer, epoch)
  File "/home/felixwagner/MEGAsync/Projektarbeit/Code/lib/utils_ODEMultistep.py", line 27, in train
    stop=i)
  File "/home/felixwagner/miniconda3/envs/felix_ml/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/felixwagner/MEGAsync/Projektarbeit/Code/lib/utils_ODEMultistep.py", line 101, in forward
    td -= ni_pred[:, l].pow(2)

Traceback (most recent call last):
  File "/home/felixwagner/MEGAsync/Projektarbeit/Code/train_ODEMultistep.py", line 164, in <module>
    train(args, model, criterion, train_loader, optimizer, epoch)
  File "/home/felixwagner/MEGAsync/Projektarbeit/Code/lib/utils_ODEMultistep.py", line 36, in train
    loss.backward()
  File "/home/felixwagner/miniconda3/envs/felix_ml/lib/python3.7/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/felixwagner/miniconda3/envs/felix_ml/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1]], which is output 0 of SelectBackward, is at version 3; expected version 2 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Somehow the gradient gets destroyed when I pow(2) the outputs of the net and then re-insert it for computing the next step in the differential equation. Can somebody help me finding a solution to this problem? Can I do anything about it?