My test code is:
                if phase == 'train':
                    loss.backward()
                    print(metric.weight.grad[0][:10])

                    a = metric.weight.data.clone()
                    print(a[0][:10])
                    optimizer.step()
                    b = metric.weight.data.clone()
                    print(b[0][:10])
                    equal = abs(a-b)
                    print(torch.sum(equal > 1e-7))  

output is:
tensor([-1.4628e-09, -7.1153e-09, -3.9151e-09, -4.6540e-09,  6.4179e-09,
        -3.9719e-09, -5.8276e-09, -7.8870e-09, -5.6572e-09,  4.5232e-09],
       device='cuda:0') # metric.weight.grad
tensor([ 0.0327, -0.0318,  0.0316, -0.0522, -0.0627,  0.0217,  0.0545,  0.0484,
         0.0454,  0.0652], device='cuda:0') # weight before update
tensor([ 0.0323, -0.0314,  0.0312, -0.0515, -0.0618,  0.0214,  0.0537,  0.0477,
         0.0448,  0.0644], device='cuda:0') # weight after update
tensor(359384, device='cuda:0')

I found some mistakes in my test code, and my weights are updated actually. But I still get a constant loss and accuracy.