Solved by ptrblck in post #2 
              
              
                SWA was added into PyTorch 1.6 in this PR, so it looks as if the contrib implementation was merged into the core.  
EDIT: Here is also the blog post.