<p dir="auto">System information</p>
<p dir="auto">Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes<br>
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 20.04<br>
I have implemented a classification model , which has Batchnormalization layers into it. While training the model , i used <code class="notranslate">tf.keras.layers.BatchNormalization</code> layer for training. Then manually extracted the bn weights (gamma, beta, mean , var) from checkpoint file and called the bn using <code class="notranslate">tf.nn.batch_normalization</code> layer using above statistics for inference. I observed a drop in testing accuracy (~75%) as compared to the scenario where i load the weights in the model using keras itself model.load_weights(chkpnt_file) (~92%).<br>
Though if i replace the <code class="notranslate">tf.nn.batch_normalization</code> call to keras bn layer call by calling set_weights function , both the cases gived 0.0 diff, which apprently led me to believe that there is diff in both the calls while the input and bn_weights are exactly same. epsilon is also same.</p>
<p dir="auto">Standalone code to reproduce the issue<br>
Please refer to this colab notebook(<a href="https://colab.research.google.com/drive/1nysYyCR5Ay_jkPlh1WD9A1Mf01eVvhMu" rel="nofollow">https://colab.research.google.com/drive/1nysYyCR5Ay_jkPlh1WD9A1Mf01eVvhMu</a>? usp=sharing) for a small example. Here the error is coming to be a margin of 10^-3 which theoretically shouldn't be there.</p>
<p dir="auto">Hi Everyone,</p>
<p dir="auto">It seems TimeDistributed wrapper forces a single shape on data unlike Recurrent layers. To reuse the Dense layer you need to create 2 TimeDistributed layers that reference the same Dense layer, otherwise you get a shape error. I am not entirely sure this behavior is desired since it duplicates the weights in the network. Here is a minimal example that should illustrate this behavior:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="#Breaks 
import keras.layers as L

shared=L.TimeDistributed(L.Dense(1))

inp1=L.Input((2,1))
inp2=L.Input((3,1))

hidden1=shared(inp1)
hidden2=shared(inp2)"><pre class="notranslate"><code class="notranslate">#Breaks 
import keras.layers as L

shared=L.TimeDistributed(L.Dense(1))

inp1=L.Input((2,1))
inp2=L.Input((3,1))

hidden1=shared(inp1)
hidden2=shared(inp2)
</code></pre></div>
<p dir="auto">Here is the error:</p>
<blockquote>
<p dir="auto">ValueError: Input 0 is incompatible with layer time_distributed_19: expected shape=(None, 2, 1), found shape=(None, 3, 1)</p>
</blockquote>
<p dir="auto">As you can see Keras 'remembered the shape of the inp1 and threw an error when found mismatch</p>
<p dir="auto">This on the other hand works fine:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="#Works but duplicates the share layer weights and unnecessarily consumes memory:
shared=(L.Dense(1))

inp1=L.Input((2,1))
inp2=L.Input((3,1))

hidden1=L.TimeDistributed(shared)(inp1)
hidden2=L.TimeDistributed(shared)(inp2)"><pre class="notranslate"><code class="notranslate">#Works but duplicates the share layer weights and unnecessarily consumes memory:
shared=(L.Dense(1))

inp1=L.Input((2,1))
inp2=L.Input((3,1))

hidden1=L.TimeDistributed(shared)(inp1)
hidden2=L.TimeDistributed(shared)(inp2)
</code></pre></div>
<p dir="auto">Now the same logic but for the Recurrent layer works without any issue even with the wrapper :</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="#Works without layer duplication
shared=L.Bidirectional(L.GRU(1))

inp1=L.Input((2,1))
inp2=L.Input((3,1))

hidden1=shared(inp1)
hidden2=shared(inp2)"><pre class="notranslate"><code class="notranslate">#Works without layer duplication
shared=L.Bidirectional(L.GRU(1))

inp1=L.Input((2,1))
inp2=L.Input((3,1))

hidden1=shared(inp1)
hidden2=shared(inp2)
</code></pre></div>
<p dir="auto">Please let me know your thoughts or if I should provide more detail.</p>
<p dir="auto">Thanks,<br>
Tim</p>
<ul dir="auto">
<li>
<p dir="auto">[ x] Check that you are up-to-date with the master branch of Keras. You can update with:<br>
pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps</p>
</li>
<li>
<p dir="auto">[ x] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found <a href="https://www.tensorflow.org/get_started/os_setup" rel="nofollow">here</a>.</p>
</li>
<li>
<p dir="auto">[x ] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:<br>
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps</p>
</li>
<li>
<p dir="auto">[x ] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).</p>
</li>
</ul>