Body: <p>I modified resnet50 architecture to get a regression network. I just add batchnorm1d and ReLU layers just before the fully connected layer. During the training, the output of batchnorm1d layer is nearly equal to 3 and this gives good results for training. However, during inference, output of batchnorm1d layer is about 30 so this leads to too low accuracy for test results. In other words, outputs of batchnorm1d layers give very different normalized output during training and inference.</p>
  
  <p>What is the reason for this situation, and how can I solve it? I am using PyTorch.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-30T16:31:40.300Z
Comments.Text: What's your batch size? BN is known to work poorly with small batch sizes.
Comments.UserDisplayName: yhenon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-31T06:45:30.803Z
Comments.Text: batch size is 16
Comments.UserDisplayName: Bedrick Kiq
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-01T14:03:58.870Z
Comments.Text: are you confident you are applying the same pre-processing in training and in testing?
Comments.UserDisplayName: yhenon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-03T07:34:27.550Z
Comments.Text: yes, ı used exactly same data for training and testing. The only difference between them is batch size.
Comments.UserDisplayName: Bedrick Kiq
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-03T16:24:49.393Z
Comments.Text: did you shuffle your batches properly? meaning samples of the same class aren't always in the same mini-batch
Comments.UserDisplayName: yhenon
Tags: <convolutional-neural-networks><pytorch><batch-normalization><inference>
Title: Why does the BatchNormalization layer produce different outputs during training and inference?
UserDisplayName: Bedrick Kiq
ThreadText: # Why does the BatchNormalization layer produce different outputs during training and inference?
  
  (Asked by: username_0 on 2020-01-30)
  
  I modified resnet50 architecture to get a regression network. I just add batchnorm1d and ReLU layers just before the fully connected layer. During the training, the output of batchnorm1d layer is nearly equal to 3 and this gives good results for training. However, during inference, output of batchnorm1d layer is about 30 so this leads to too low accuracy for test results. In other words, outputs of batchnorm1d layers give very different normalized output during training and inference.
  
  What is the reason for this situation, and how can I solve it? I am using PyTorch.
  
  **Tags:** convolutional-neural-networks, pytorch, batch-normalization, inference
  > **Comments:**
  > **username_1**: What's your batch size? BN is known to work poorly with small batch sizes.  
  > **username_0**: batch size is 16  
  > **username_1**: are you confident you are applying the same pre-processing in training and in testing?  
  > **username_0**: yes, ı used exactly same data for training and testing. The only difference between them is batch size.  
  > **username_1**: did you shuffle your batches properly? meaning samples of the same class aren't always in the same mini-batch  
  
  ---
  
  ## 0 answers
  
  