I think your code is correct and the initial “checkpoint” would be created after swa_start epochs were already done. Afterwards the update would take place.
I’m unsure, if this is “necessary” or if the posted example would also work fine, since the initial updates would become less important during the training.
Did you run your code and saw a difference using both approaches?
If so, which one performed better?