hello,
In the past few days i tried to rebuild an autograd engine in the style of pytorch.
All of the computed gradients do actually match  pytorch’s gradients so it should all be fine.
Now i decided to build a little nn library ontop of my autograd engine to train some small networks.
So my problem is that due to operations like the .mean() in the end the gradients get scaled down to basically nothing so it takes a huge amount of optimizer steps to learn anything. So my question is if
pytorch somehow scales gradients when optimizing parameters and if so how?
have a nice day