I’m currently working with the 3detr repo(https://github.com/facebookresearch/3detr), and it is only officially working with Pytorch 1.9. When I switched to Pytorch 1.10, I got an error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [256, 1, 256]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)
Turning on anomaly detection, I manage to trace errors to dropout layers, which I changed to work with Pytorch 1.10, like this :
self.dropout = nn.Dropout(dropout, inplace=False) # Inplace Originally True, set to False for Pytorch 1.10 Compatibility
Since I posting large sections of the 3dter repo won’t be easy to read(and I would have to post a lot of code), instead, I created a pull request on the 3detr repo with the changes: (Pull Request Here 1)
However, the dropout layers would returns errors on lines like this: src2 = self.linear2(self.dropout(self.activation(self.linear1(src2))))
My first question is what changed between Pytorch 1.9 and Pytorch 1.10 that would cause errors like this to occur in Pytorch 1.10, but not in 1.9?
My second question is however would I fix this errors without setting inplace to False(this is undesirable for me since it increases train time per batch by around 2x!) or rolling back to 1.9, and if this is not possible to do, is there any way to ignore the error Runtime Errors?