I found the problem in my code, it’s because of the cudnn batch norm. According to this github issue 164, the solution is to edit the batchnorm part in torch/nn/functional.py or set torch.backends.cudnn.enabled = False.