Solved by albanD in post #2 
              
              
                The thing is that the backward operation of masked_fill() needs to know where the input was written into. So it needs the value of the mask. So if you modify it inplace, the autograd engine will fail to compute gradients.