I tried Adam and AdamW with varying learning rates from a range between 10**-2 to 10**-6, and varying weight decays.
turns out that in my case an initial learning rate of 10**-4 seems to be the best solution.
moreover, i use a learning rate scheduler that lowers the learning rate via lr_i = 0.9*lr_{i-1} but it doesn’t seem to have an impact on the model performance and overfitting