@sgugger One thing I noticed, doing other experiments, is that using the optimizer as I have implemented it, after some epochs the learning rate reaches 0.0, while with the default one I always have a value > 0.
I don’t know if this information can be used to understand how my implementation differs from the default one. They look really identical to me … I have no idea…