Hey guys, recent I was quite puzzled about the learning rate for Adam optimizer in Pytorch:
In many demos I have read, it seems that the proper value of lr for Adam is usually 0.01 or 0.001. However, when I set lr=0.001 in my code, the training loss and accuracy will increasing and decreasing violently all the time,  and never converge at all.
So, only when I set lr around 0.00001, the training process becomes to be  normal, and finally my model works well ! Though the problem has been solved, I think the current value of lr is still too much small =_=.
Is this phenomenon normal? If not, what caused it? I am waiting sincerely for you guys to solve my puzzle!!!