I was planning to add SWA to transformers by working on a PR. After seeing this thread, I am a lot skeptical about starting working on it. @sgugger I have some questions for you. Any answer will be much appreciated.

You have said that it works well with cyclical learning rates. When I look at the paper 3 and also the image below, I see that constant learning rate also achieves better results.
Did you experiment transformers with SWA and constant learning rate?

@prajjwal1 Do you have any other insights about SWA with transformers?
FYI: I also opened a feature request 9 to get more insights from others.