I am really confused about choosing the best learning rate and weight decay. I conducted an ablation study and got the following results:


For both experiments a weight decay = 0.001 was used.


For the first plot: lr = 0.003  => best validation loss: 0.405 reached in epoch 4, train_loss = 0.326 at same epoch


For the second plot: lr = 0.03 => best validation loss: 0.403 reached in epoch 9, train_loss = 0.411 also at same epoch



lr0,003-wd0,0011848×977 47.6 KB


lr0,03-wd0,0011848×977 53.6 KB