Pytorch 1.2 on linux with cuda 10.0.
Got the tracing info by applying torch.autograd.set_detect_anomaly(True).
The error shows the
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [2, 1, 1024]] is at version 1; expected version 0 instead. Hint: the backtrace further above shows       the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

and it points to
states = hidden_states.gather(-2, positions)

Any ideas? Thanks.