Sorry, my fault. This problem is not related to Pytorch itself. I came across numerical instability when I used ‘lr=0’.    'requires_grad = False’ and ‘lr = 0’ do output same results. BTY, ‘lr=0’ requires more computation.
Thank you.