I changed all the -=, += to the not-inplace version, but I get the same error at the same line:
sys:1: RuntimeWarning: Traceback of forward call that caused the error:
  File "/home/felixwagner/MEGAsync/Projektarbeit/Code/train_ODEMultistep.py", line 164, in <module>
    train(args, model, criterion, train_loader, optimizer, epoch)
  File "/home/felixwagner/MEGAsync/Projektarbeit/Code/lib/utils_ODEMultistep.py", line 27, in train
    stop=i)
  File "/home/felixwagner/miniconda3/envs/felix_ml/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/felixwagner/MEGAsync/Projektarbeit/Code/lib/utils_ODEMultistep.py", line 101, in forward
    td = td - ni_pred[:, l].pow(2)

Traceback (most recent call last):
  File "/home/felixwagner/MEGAsync/Projektarbeit/Code/train_ODEMultistep.py", line 164, in <module>
    train(args, model, criterion, train_loader, optimizer, epoch)
  File "/home/felixwagner/MEGAsync/Projektarbeit/Code/lib/utils_ODEMultistep.py", line 36, in train
    loss.backward()
  File "/home/felixwagner/miniconda3/envs/felix_ml/lib/python3.7/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/felixwagner/miniconda3/envs/felix_ml/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1]], which is output 0 of SelectBackward, is at version 3; expected version 2 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!


I also tried to do multiple of the calculations in one line, instead of only one per line, which gives the same error (not surprisingly).