I am working on an unusual research project and need to modify some gradients in-place.
This worked fine so far, but I just updated to a newer version of Pytorch, and now I am getting this error:
“RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [762, 5]], which is output 0 of TBackward, is at version 5; expected version 2 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!”
I get that this is an error for almost every usecase, but I actually want to do this on purpose.
Is there a way to turn this error off?
Alternatively, is there another way to modify gradients that is legal / preferred?
(My situation: I’m running a network where the architecture is not fixed but changes based on intermediate results. To backpropagate through this safely, I break the network into smaller parts, and manually retrieve and re-insert the gradients at the appropriate tensors.)