Putting module.eval() works and does not change the performance.
But the bn layer keep changning running_mean and running_var during training even though I freezed them with module.eval().
So I thought I should also turn the track_running_stats off to prevent the running_mean and running_var which is used in inference.