title: Keras BatchNorm: Training accuracy increases while Testing accuracy decreases
body: <p>I am trying to use BatchNorm in Keras. The training accuracy increases over time. From 12% to 20%, slowly but surely.
  The test accuracy however decreases from 12% to 0%. Random baseline is 12%.</p>
  
  <p>I very much assume this is due to the batchnorm layer (removing the batchnorm layer results in ~12% test accuracy), which maybe does not initialize parameters gamma and beta well enough. Do I have to regard anything special when applying batchnorm? I don't really understand what else could have gone wrong. I have the following model:</p>
  
  <p>model = Sequential()</p>
  
  <pre><code>model.add(BatchNormalization(input_shape=(16, 8)))
  model.add(Reshape((16, 8, 1)))
  
  #1. Conv (64 filters; 3x3 kernel)
  model.add(default_Conv2D())
  model.add(BatchNormalization(axis=3))
  model.add(Activation('relu'))
  
  #2. Conv (64 filters; 3x3 kernel)
  model.add(default_Conv2D())
  model.add(BatchNormalization(axis=3))
  model.add(Activation('relu'))
  
  ... 
  
  #8. Affine (NUM_GESTURES units) Output layer
  model.add(default_Dense(NUM_GESTURES))
  model.add(Activation('softmax'))
  
  
  sgd = optimizers.SGD(lr=0.1)
  model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
  </code></pre>
  
  <p>default_Conv2D and default_Dense are defined as follows:</p>
  
  <pre><code>def default_Conv2D():
      return Conv2D(
          filters=64,
          kernel_size=3,
          strides=1,
          padding='same',
          # activation=None,
          # use_bias=True,
          # kernel_initializer=RandomNormal(mean=0.0, stddev=0.01, seed=None), #RandomUniform(),
          kernel_regularizer=regularizers.l2(0.0001),
          # bias_initializer=RandomNormal(mean=0.0, stddev=0.01, seed=None), # RandomUniform(),
          # bias_regularizer=None
      )
  
  def default_Dense(units):
  
      return Dense(
          units=units,
          # activation=None,
          # use_bias=True,
          # kernel_initializer=RandomNormal(mean=0.0, stddev=0.01, seed=None),#RandomUniform(),
          # bias_initializer=RandomNormal(mean=0.0, stddev=0.01, seed=None),#RandomUniform(),
          kernel_regularizer=regularizers.l2(0.0001),
          # bias_regularizer=None
      )
  </code></pre>