Hi, Thanks for replying!!!
I am afraid the difference is beyond tolerance after about 7 iters. Here is the log message of the first 10 iters, the difference is quite smaller in the first iter, but the it grows rapidly in the following few iters:
grad2:  tensor([ 6.1093e-05,  3.2695e-05, -9.3787e-05], device='cuda:0')
feat.grad1 tensor([ 2.6212e-06,  1.7502e-06,  1.8138e-06, -2.5879e-06], device='cuda:0')
feat.grad2 tensor([ 2.6212e-06,  1.7503e-06,  1.8139e-06, -2.5879e-06], device='cuda:0')
1.1920928955078125e-07

grad2:  tensor([ 7.4433e-07, -8.7096e-05,  8.6352e-05], device='cuda:0')
feat.grad1 tensor([-1.7985e-06,  1.4837e-06,  1.2604e-06, -9.5811e-07], device='cuda:0')
feat.grad2 tensor([-1.7828e-06,  1.4885e-06,  1.2585e-06, -9.3544e-07], device='cuda:0')
-9.5367431640625e-07

grad2:  tensor([-2.3506e-05,  2.1586e-05,  1.9193e-06], device='cuda:0')
feat.grad1 tensor([ 8.6845e-07, -6.5791e-07,  3.0064e-06,  5.4995e-06], device='cuda:0')
feat.grad2 tensor([ 1.3321e-06, -1.5314e-07,  2.5235e-06,  4.8529e-06], device='cuda:0')
-0.00011467933654785156

grad2:  tensor([ 5.4759e-08,  2.2978e-06, -2.3525e-06], device='cuda:0')
feat.grad1 tensor([-3.3766e-06, -2.7148e-06,  2.5728e-06, -3.7042e-06], device='cuda:0')
feat.grad2 tensor([-2.0076e-06, -1.5157e-06,  3.2242e-06, -3.4447e-06], device='cuda:0')
-0.013294696807861328

grad2:  tensor([ 5.9485e-07, -5.9485e-07,  9.8629e-14], device='cuda:0')
feat.grad1 tensor([-5.6624e-06, -8.5295e-08,  5.8959e-07, -1.6566e-06], device='cuda:0')
feat.grad2 tensor([-3.1079e-06,  3.5109e-06,  2.4890e-06, -2.3740e-06], device='cuda:0')
-0.04217672348022461

grad2:  tensor([ 1.2207e-04, -1.2207e-04,  1.4023e-10], device='cuda:0')
feat.grad1 tensor([-2.6114e-06, -2.7196e-06,  2.5282e-06, -1.0882e-06], device='cuda:0')
feat.grad2 tensor([-3.4735e-06,  2.6183e-06,  3.2230e-06, -1.0301e-06], device='cuda:0')
-0.02991771697998047

grad2:  tensor([-1.2201e-04,  9.0557e-05,  3.1458e-05], device='cuda:0')
feat.grad1 tensor([ 7.7824e-07,  1.6687e-06, -1.0701e-06,  1.8441e-06], device='cuda:0')
feat.grad2 tensor([ 7.6660e-07, -5.4422e-08, -3.0931e-06, -1.6282e-06], device='cuda:0')
0.05857133865356445

grad2:  tensor([-2.1591e-05,  1.8860e-05,  2.7316e-06], device='cuda:0')
feat.grad1 tensor([-1.5478e-06, -1.0981e-06,  2.9503e-06, -7.7435e-07], device='cuda:0')
feat.grad2 tensor([ 1.4916e-06, -1.2717e-06, -1.8291e-07,  4.2905e-06], device='cuda:0')
-0.11451804637908936

grad2:  tensor([ 5.3404e-10,  1.0895e-04, -1.0895e-04], device='cuda:0')
feat.grad1 tensor([-2.5790e-06, -6.7249e-08, -2.2561e-06, -2.0569e-06], device='cuda:0')
feat.grad2 tensor([-2.1519e-06,  8.4891e-07,  8.0614e-07, -1.6089e-06], device='cuda:0')
0.4443323612213135

grad2:  tensor([ 7.9950e-07, -1.2178e-04,  1.2098e-04], device='cuda:0')
feat.grad1 tensor([ 3.2793e-06,  5.9981e-07, -4.5706e-07, -3.7950e-06], device='cuda:0')
feat.grad2 tensor([-1.7417e-06, -1.5725e-06,  1.4380e-07, -1.4676e-06], device='cuda:0')
0.5565323829650879