Standard deep learning pipeline runs a training epoch followed by a testing epoch. Assume I would like to compute some statistics during every iteration of training. Assume also that my model has batch norm and so I have to switch to eval mode every iteration. I was wondering if this would result in the statistics of batch norm being reset every iteration or not. Also, is there something inherently wrong with switching from train to eval and backwards in each iteration of an epoch.
Thanks!