There are other layers besides Batchnorm that behave differently on train and eval; or instance, dropout layer or layers with spectral norm etc. Therefore it s a very good practice to always set the model to eval when testing and to train when training