Just curious…Why these two scripts in pytorch tutorial have very different learning rates (1e-6 in autograd vs. 1e-4 in nn-module), when they’re essentially doing the same thing?
http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#autograd 11
http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#nn-module 7
I changed torch.nn.Linear(D_in, H) to torch.nn.Linear(D_in, H, bias = False) in the nn-module example,  to keep accordance with the no bias setting in the autograd example.
If I interchange the learning rate in the autograd example and nn-module example, the loss either decreases very slowly or blows up.
I noticed the initial errors are very different (about 3e7 in autograd vs. 600 in nn-module). I guess this could be the reason for different learning rate settings. But wonder why this happens?
Guess the answer is somewhat apparent, but really couldn’t find out.