how to set the init learning rate when use bertAdam?
i set the BertAdam learning rate as the default value of args (3e-5), and i step in to the BertAdam step by step , and print lr_scheduled see that the acturly lr is very small over all the training process (between <0 ~ 1> * 3e-5), this cause the loss decrease very slow, when i set the init learning rate as 0.1, the loss decrease much more fast, so what's the right way to set the learning rate for BertAdam param?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
