SWA AveragedModel proper usage
Hi, I want to use SWA technique and finally is official at Pytorch! https://pytorch.org/blog/pytorch-1.6-now-includes-stochastic-weight-averaging/ 8
But there is something not clear. In the example they post, the averagemodel is created out of the initial training loop. My question is… If we want to average from last model weights, as should be and not initial random weights, we should create the averagemodel just before swa starts, something like this (following example nomenclature):
if (epoch+1) == swa_start:
    swa_model = AveragedModel(model)