Some update:
It seems the problem is that the forward and backward computation of Pytorch is not thread-safe. I’m not sure whether this is the problem introduced by Autograd. Can anyone confirm whether PyTorch is thread-safe or not?