Solved by SimonW in post #2 
              
              
                In train mode, you are using batch stats (not running stats). Since you disabled running stats update, in eval mode, the running stats are still the old stats, and since eval mode uses these old running stats, results will be bad.