Yes, I wouldn’t use the validation dataset. What I meant is to update the BN stats using the training set before running validation, similar to the code example above (loader is the training set).
Ok so looking at swa/train.py at 411b2fcad59bec60c6c9eb1eb19ab906540e5ea2 · timgaripov/swa · GitHub 9 I think I was right in that we need to update the BN stats of the swa_model before we validate. Relevant lines 158, 159 @hktxt @ptrblck