@chenyuntc
I see some improvements but still far away from Numpy’s. I set the random seed to 10000 and tried to print the weights in each epoch. I could not understand two specific behaviors.

Numpy weights are 8 decimal valued (0.65897509) while pytorch is rounding it to 4 digit (0.6590).
The change in weight values are much faster in numpy

Change of first first weight matrix’s first value in Numpy

0.65897509416461408
0.63824423351700321
0.70742434009324673
0.74590493637361743
0.76070011091500567
0.76704695584086358
0.77166554222431838
0.77654595399345228
0.78221473000011987
0.7887668125254953

Change of first first weight matrix’s first value in Pytorch

0.6590
0.6585
0.6580
0.6575
0.6570
0.6565
0.6561
0.6556
0.6551
0.6547
0.6542

Since both script using seed(1000) we got the same initial value (pytorch rounded it though) but the change is super slow in pytorch