What do you mean by “does work?” you clearly say that it made your results worse.
From what I saw over T5-small “scale parameter” makes the result slightly worse as well.
I’m currently trying T5-3B and the model doesn’t learn
(beta1=0.9 and scale_parameter=False) and default learning rate, so I wonder what can be done to mitigate it.