It still cannot solve it.
Are there any wrong in me check process described below?
I ran my code twice on GPU, after optimizer.step() I used torch.save(model[1]._parameters['weight'].cpu().data, 'w1') to save the weight to my disk (the second time changes w1 to w2). Then I loaded the two weights using torch.load('w1') and torch.load('w2'), subtracting them and check if the results are all 0.