Thanks for the solution.
So if I want to use the individual batch statistics for the validation (during the training process), and to obtain the statistics over the dataset in the end of the training, is the following process correct?
1, set “track_running_stats” to true, allow for keeping the average of the statistics.
2, train the model
3, set the model to .val(), but set batch norm to .train(), to validate the model using individual statistics? This does not work since it will update the statistics using the validation data. One trick is to set the saved average statistics to its initial state (0 or 1) before the final updating, so the data does not show effect on the final statistics. Is there any easy way?
4, set the model to .train(), but don’t do backward pass to estimate the statistics for the whole dataset. (here the statistics is changed from the average in the training. Any way to avoid this?)
How to correctly implement “use the individual batch statistics for the validation (during the training process), and to obtain the statistics over the dataset in the end of the training”?
Is there any way to set “track_running_stats” during the training processing similarly as .train() and .val() option?
Thanks.