I see no reason why it wouldn’t work. Since SWA holds a separate copy of the model on each rank and uses the same all-reduced gradients to update the weight averages, it should theoretically work.
I would suggest giving it a try, and letting us know if you experience any issues.