title: Batch Normalization in tf.keras does not calculate average mean and average variance
body: <p>A similar unanswered question was asked <a href="https://stackoverflow.com/questions/50164572/batchnormalization-in-keras">here</a>.
  I am testing one deep reinforcement learning algorithm which uses keras backend in tensorflow. I am not very familiar with tf.keras, nevertheless would like to add batch normalization layers. Therefore, I am trying to use <code>tf.keras.layers.BatchNormalization()</code>, but it does not update average means and variances because <code>update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)</code> is empty. </p>
  
  <p>Using the regular <code>tf.layers.batch_normalization</code> seem to work fine. However, because the complete algorithm is somewhat complicated, <strong>I need to find a way to use <code>tf.keras</code></strong>.</p>
  
  <p>A standard <code>tf</code> layer <code>batch_normed = tf.layers.batch_normalization(hidden, training=True)</code> updates the averages since <code>update_ops</code> is not empty:</p>
  
  <pre><code>[
      &lt;tf.Operation 'batch_normalization/AssignMovingAvg' type=AssignSub&gt;, 
      &lt;tf.Operation 'batch_normalization/AssignMovingAvg_1' type=AssignSub&gt;, 
      &lt;tf.Operation 'batch_normalization_1/AssignMovingAvg' type=AssignSub&gt;, 
      &lt;tf.Operation 'batch_normalization_1/AssignMovingAvg_1' type=AssignSub&gt;
  ]
  </code></pre>
  
  <p>Minimal example that does <strong>not</strong> work:</p>
  
  <pre><code>import tensorflow as tf
  import numpy as np
  
  tf.reset_default_graph()
  graph = tf.get_default_graph()
  tf.keras.backend.set_learning_phase(True)
  
  input_shapes = [(3, )]
  hidden_layer_sizes = [16, 16]
  
  inputs = [
      tf.keras.layers.Input(shape=input_shape)
      for input_shape in input_shapes
  ]
  
  concatenated = tf.keras.layers.Lambda(
      lambda x: tf.concat(x, axis=-1)
  )(inputs)
  
  out = concatenated
  for units in hidden_layer_sizes:      
      hidden = tf.keras.layers.Dense(
      units, activation=None
      )(out)
      batch_normed = tf.keras.layers.BatchNormalization()(hidden, training=True)
      #batch_normed = tf.layers.batch_normalization(hidden, training=True)
      out = tf.keras.layers.Activation('relu')(batch_normed)
  
  out = tf.keras.layers.Dense(
      units=1, activation='linear'
  )(out)
  
  
  data = np.random.rand(100,3)
  with tf.Session(graph=graph) as sess:
      sess.run(tf.global_variables_initializer())
  
      for i in range(10):
  
      update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
  
      sess.run(update_ops,  {inputs[0]: data})
      sess.run(out, {inputs[0]: data})
  
      variables = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,
                            scope='batch_normalization')
      bn_gamma, bn_beta, bn_moving_mean, bn_moving_variance = [], [], [], []
      for variable in variables:
          val = sess.run(variable)
          nv = np.linalg.norm(val)
          if 'gamma' in variable.name:
              bn_gamma.append(nv)
          if 'beta' in variable.name:
              bn_beta.append(nv)
          if 'moving_mean' in variable.name:
              bn_moving_mean.append(nv)
          if 'moving_variance' in variable.name:
              bn_moving_variance.append(nv)
  
          diagnostics = {
              'bn_Q_gamma': np.mean(bn_gamma),
              'bn_Q_beta': np.mean(bn_beta),
              'bn_Q_moving_mean': np.mean(bn_moving_mean),
              'bn_Q_moving_variance': np.mean(bn_moving_variance),
          }
  
      print(diagnostics)
  </code></pre>
  
  <p>The output is the following (you can see moving_mean and moving_variance not changing):</p>
  
  <pre><code>{'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.0, 'bn_Q_moving_variance': 4.0}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.0, 'bn_Q_moving_variance': 4.0}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.0, 'bn_Q_moving_variance': 4.0}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.0, 'bn_Q_moving_variance': 4.0}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.0, 'bn_Q_moving_variance': 4.0}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.0, 'bn_Q_moving_variance': 4.0}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.0, 'bn_Q_moving_variance': 4.0}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.0, 'bn_Q_moving_variance': 4.0}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.0, 'bn_Q_moving_variance': 4.0}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.0, 'bn_Q_moving_variance': 4.0}
  </code></pre>
  
  <p>While the expected output is something like the following (comment the line with <code>batch_normed</code> calculus using <code>tf.keras</code> and uncomment the one below it):</p>
  
  <pre><code>{'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.0148749575, 'bn_Q_moving_variance': 3.966927}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.029601166, 'bn_Q_moving_variance': 3.934192}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.04418011, 'bn_Q_moving_variance': 3.9017918}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.05861327, 'bn_Q_moving_variance': 3.8697228}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.0729021, 'bn_Q_moving_variance': 3.8379822}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.08704803, 'bn_Q_moving_variance': 3.8065662}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.10105251, 'bn_Q_moving_variance': 3.7754717}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.11491694, 'bn_Q_moving_variance': 3.7446957}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.12864274, 'bn_Q_moving_variance': 3.7142346}
  {'bn_Q_gamma': 4.0, 'bn_Q_beta': 0.0, 'bn_Q_moving_mean': 0.14223127, 'bn_Q_moving_variance': 3.6840856}
  </code></pre>
  
  <h2>Note</h2>
  
  <p>There is still something fishy even with <code>tf.layers.batch_normalization</code>. The standard <code>tf</code> approach of <code>tf.control_dependencies</code>:</p>
  
  <pre><code>    with tf.control_dependencies(update_ops):
          sess.run(out, {inputs[0]: data})
  </code></pre>
  
  <p>which I place instead of the following two lines in the code above:</p>
  
  <pre><code>    sess.run(update_ops,  {inputs[0]: data})
      sess.run(out, {inputs[0]: data})
  </code></pre>
  
  <p>produces <code>bn_Q_moving_mean = 0.0</code> and <code>bn_Q_moving_variance = 4.0</code></p>