<p dir="auto">When batchnorm is used inside a model that is invoked more than once (e.g inside an auto regression step) it produces very different results during prediction (non train time). I have isolated a toy problem and the suspected reason is that the updated values are accumulated across all invocations and then updated using <code class="notranslate">K.moving_average_update</code>. In the example bellow we use momentum 0.5 on the means 2, 4, 8  then we should get <code class="notranslate">6.5 = ((0.5 * 0 + 0.5 * 2) * 0.5 + 0.5 * 4) + 0.5 * 8</code> or <code class="notranslate">2.33 = 0.5 * 0 +0.5* (2 + 4 + 8) / 3</code> but we have (with tensorflow)<br>
<code class="notranslate">7 = 0.5 * 0 + 0.5 * (2 + 4 + 8)</code> also if we use momentum=0.1 for example the output is <code class="notranslate">12.666 = 0.1 * + 0.9 * (2 + 4 +  8)</code></p>
<p dir="auto">With theano, only the first invocation is considered</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import numpy as np
from keras.models import Model
from keras.layers import Layer, Input, Dense
from keras import initializers
import keras.backend as K

class CollectAvg(Layer):
    def build(self, input_shape):
        self.moving_mean = self.add_weight(shape=(input_shape[1], ), name='moving_mean',
                                           initializer=initializers.get('zeros'),
                                           trainable=False)
        self.built = True
        
    def call(self, inputs):
        mean = K.mean(inputs)
        self.add_update([K.moving_average_update(self.moving_mean, mean, 0.5)], inputs)
        return inputs

def get_shared_model():
    x = shared_input = Input(shape=(1, ))
    
    w = np.array([2]).reshape((1, 1, 1))
    x = Dense(1, use_bias=False, weights = w)(x)
    
    x = CollectAvg()(x)
    
    shared_model = Model(inputs=[shared_input], outputs=[x]) 
    return shared_model

def get_auto_regressive(submodel, n):
    x = model_input = Input(shape=(1, ))
    outputs = []
    for i in range(n):
        x = submodel(x)
        outputs.append(x)
    
    model = Model(inputs=[model_input], outputs = outputs)
    model.compile(optimizer='sgd', loss='mse')
    return model

submodel = get_shared_model()
submodel.summary()
model = get_auto_regressive(submodel, 3)
model.summary()
sample_input = np.asarray([[1]])
for i in range(2):
    sample_output = model.predict(sample_input)
    model.train_on_batch(sample_input, sample_output)
    print (&quot;collected avg:&quot;,  submodel.layers[-1].get_weights())
print (&quot;outputs:&quot;, np.array(sample_output))"><pre class="notranslate"><code class="notranslate">import numpy as np
from keras.models import Model
from keras.layers import Layer, Input, Dense
from keras import initializers
import keras.backend as K

class CollectAvg(Layer):
    def build(self, input_shape):
        self.moving_mean = self.add_weight(shape=(input_shape[1], ), name='moving_mean',
                                           initializer=initializers.get('zeros'),
                                           trainable=False)
        self.built = True
        
    def call(self, inputs):
        mean = K.mean(inputs)
        self.add_update([K.moving_average_update(self.moving_mean, mean, 0.5)], inputs)
        return inputs

def get_shared_model():
    x = shared_input = Input(shape=(1, ))
    
    w = np.array([2]).reshape((1, 1, 1))
    x = Dense(1, use_bias=False, weights = w)(x)
    
    x = CollectAvg()(x)
    
    shared_model = Model(inputs=[shared_input], outputs=[x]) 
    return shared_model

def get_auto_regressive(submodel, n):
    x = model_input = Input(shape=(1, ))
    outputs = []
    for i in range(n):
        x = submodel(x)
        outputs.append(x)
    
    model = Model(inputs=[model_input], outputs = outputs)
    model.compile(optimizer='sgd', loss='mse')
    return model

submodel = get_shared_model()
submodel.summary()
model = get_auto_regressive(submodel, 3)
model.summary()
sample_input = np.asarray([[1]])
for i in range(2):
    sample_output = model.predict(sample_input)
    model.train_on_batch(sample_input, sample_output)
    print ("collected avg:",  submodel.layers[-1].get_weights())
print ("outputs:", np.array(sample_output))
</code></pre></div>
<p dir="auto">with output being</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 1)                 0         
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 1         
_________________________________________________________________
collect_avg_1 (CollectAvg)   (None, 1)                 1         
=================================================================
Total params: 2
Trainable params: 1
Non-trainable params: 1
_________________________________________________________________
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         (None, 1)                 0         
_________________________________________________________________
model_1 (Model)              (None, 1)                 2         
=================================================================
Total params: 2
Trainable params: 1
Non-trainable params: 1
_________________________________________________________________"><pre class="notranslate"><code class="notranslate">Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 1)                 0         
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 1         
_________________________________________________________________
collect_avg_1 (CollectAvg)   (None, 1)                 1         
=================================================================
Total params: 2
Trainable params: 1
Non-trainable params: 1
_________________________________________________________________
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         (None, 1)                 0         
_________________________________________________________________
model_1 (Model)              (None, 1)                 2         
=================================================================
Total params: 2
Trainable params: 1
Non-trainable params: 1
_________________________________________________________________
</code></pre></div>
<p dir="auto">for theano:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="collected avg: [array([ 1.], dtype=float32)]
collected avg: [array([ 1.5], dtype=float32)]
outputs: [[[ 2.]]

 [[ 4.]]

 [[ 8.]]]"><pre class="notranslate"><code class="notranslate">collected avg: [array([ 1.], dtype=float32)]
collected avg: [array([ 1.5], dtype=float32)]
outputs: [[[ 2.]]

 [[ 4.]]

 [[ 8.]]]
</code></pre></div>
<p dir="auto">for tensorflow</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="collected avg: [array([ 7.], dtype=float32)]
collected avg: [array([ 3.5], dtype=float32)]
outputs: [[[ 2.]]

 [[ 4.]]

 [[ 8.]]]"><pre class="notranslate"><code class="notranslate">collected avg: [array([ 7.], dtype=float32)]
collected avg: [array([ 3.5], dtype=float32)]
outputs: [[[ 2.]]

 [[ 4.]]

 [[ 8.]]]
</code></pre></div>
<p dir="auto">Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on <a href="http://stackoverflow.com/questions/tagged/keras" rel="nofollow">StackOverflow</a> or <a href="https://keras-slack-autojoin.herokuapp.com/" rel="nofollow">join the Keras Slack channel</a> and ask there instead of filing a GitHub issue.</p>
<p dir="auto">Thank you!</p>
<ul class="contains-task-list">
<li class="task-list-item">
<p dir="auto"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Check that you are up-to-date with the master branch of Keras. You can update with:<br>
pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps</p>
</li>
<li class="task-list-item">
<p dir="auto"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found <a href="https://www.tensorflow.org/get_started/os_setup" rel="nofollow">here</a>.</p>
</li>
<li class="task-list-item">
<p dir="auto"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:<br>
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps</p>
</li>
<li class="task-list-item">
<p dir="auto"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).</p>
</li>
</ul>
<p dir="auto">This is a duplicate of: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="243590522" data-permission-text="Title is private" data-url="https://github.com/keras-team/keras/issues/7365" data-hovercard-type="issue" data-hovercard-url="/keras-team/keras/issues/7365/hovercard" href="https://github.com/keras-team/keras/issues/7365">#7365</a>.</p>
<p dir="auto"><code class="notranslate">Bidirectional.call(self, inputs, training=None, mask=None):</code> does not accept an <code class="notranslate">initial_state</code> keyword argument.</p>
<p dir="auto">Furthermore, invoking <code class="notranslate">Bidirectional.call</code> with a list of inputs doesn't allow you to specify the initial state either, since <code class="notranslate">Recurrent.call</code> only has this code to handle list inputs:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="if isinstance(inputs, list):
    inputs = inputs[0]"><pre class="notranslate"><code class="notranslate">if isinstance(inputs, list):
    inputs = inputs[0]
</code></pre></div>
<p dir="auto">I would fix this myself but I'm not sure what the right solution is. My guess is to switch <code class="notranslate">Bidirectional</code> to use <code class="notranslate">self.forward_layer.__call__</code> instead of <code class="notranslate">self.forward_layer.call</code>.</p>