Found what was the reason. In literature, the learning rates I found for fine tuning tasks are mostly mentioned in the ranges of e-4, e-5. Hence, I’ve been using 5e-5 etc. and thought that the code setup had the error. But it seemed that even lower learning rate (5e-6) was needed for this.