I did like what you did, and it seems you are right.
I checked right after the swa_model = AveragedModel(model), the weights of the swa_model parameters and model parameters are exactly the same. Doing swa_model = AveragedModel(model) before the loop starts is not a good idea.