But the entries inside the tensor got .grad func. I thought that its better to work on the tensor level in order to benefit from GPU performance (matrix multiplication) but it seems that I’m wrong. Why then each of the entires in the tensor has .grad func ?
I did the changes that u suggested, now indeed I see that the grads are set and not None. However , it seems that now I’m getting overfitting very fast and my test loss increases 
[1] train loss: 2.105
[1] test loss: 2.390
[2] train loss: 1.235
[2] test loss: 5.240
[3] train loss: 0.511
[3] test loss: 7.921
[4] train loss: 0.167
[4] test loss: 10.697
[5] train loss: 0.033
[5] test loss: 11.992
and so on…