<p dir="auto">Recently introduced batch renormalization could be very useful, but without support for a fused batch norm, it has limited use (XLA JIT still has too many problems to be used instead of a fused version)</p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow):<br>
Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):<br>
MacOSX 10.13.6</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:<br>
No</li>
<li>TensorFlow installed from (source or binary):<br>
Binary</li>
<li>TensorFlow version (use command below):<br>
tf.version.VERSION=2.0.0-dev20190404<br>
tf.version.GIT_VERSION=v1.12.0-11729-g98c3cfbf74</li>
<li>Python version:<br>
3.6.8</li>
<li>Bazel version (if compiling from source):<br>
N/A</li>
<li>GCC/Compiler version (if compiling from source):<br>
N/A</li>
<li>CUDA/cuDNN version:<br>
N/A</li>
<li>GPU model and memory:<br>
N/A</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
When using <code class="notranslate">a + b</code> then <code class="notranslate">keras.layers.Add</code> in the same Keras model, I get an exception saying the name <code class="notranslate">"add"</code> was used twice. The problem goes away if I use <code class="notranslate">keras.layers.Add</code> first, or if I use twice the same operation (either <code class="notranslate">+</code> twice, or <code class="notranslate">keras.layers.Add</code> twice).</p>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
I don't expect any name conflicts.</p>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
The following code raises a <code class="notranslate">ValueError</code> exception (full stacktrace below).</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from tensorflow import keras

inputs = keras.layers.Input(shape=[2])
add1 = inputs + inputs
add2 = keras.layers.Add()([inputs, inputs])
model = keras.models.Model(inputs=[inputs], outputs=[add1, add2])"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">import</span> <span class="pl-s1">keras</span>

<span class="pl-s1">inputs</span> <span class="pl-c1">=</span> <span class="pl-s1">keras</span>.<span class="pl-s1">layers</span>.<span class="pl-v">Input</span>(<span class="pl-s1">shape</span><span class="pl-c1">=</span>[<span class="pl-c1">2</span>])
<span class="pl-s1">add1</span> <span class="pl-c1">=</span> <span class="pl-s1">inputs</span> <span class="pl-c1">+</span> <span class="pl-s1">inputs</span>
<span class="pl-s1">add2</span> <span class="pl-c1">=</span> <span class="pl-s1">keras</span>.<span class="pl-s1">layers</span>.<span class="pl-v">Add</span>()([<span class="pl-s1">inputs</span>, <span class="pl-s1">inputs</span>])
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-s1">keras</span>.<span class="pl-s1">models</span>.<span class="pl-v">Model</span>(<span class="pl-s1">inputs</span><span class="pl-c1">=</span>[<span class="pl-s1">inputs</span>], <span class="pl-s1">outputs</span><span class="pl-c1">=</span>[<span class="pl-s1">add1</span>, <span class="pl-s1">add2</span>])</pre></div>
<p dir="auto"><strong>Other info / logs</strong><br>
Here is the full stacktrace:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ValueError                                Traceback (most recent call last)
&lt;ipython-input-1-07b5faf0201d&gt; in &lt;module&gt;
      4 add1 = inputs + inputs
      5 add2 = keras.layers.Add()([inputs, inputs])
----&gt; 6 model = keras.models.Model(inputs=[inputs], outputs=[add1, add2])

.../tensorflow/python/keras/engine/training.py in __init__(self, *args, **kwargs)
    121
    122   def __init__(self, *args, **kwargs):
--&gt; 123     super(Model, self).__init__(*args, **kwargs)
    124     # initializing _distribution_strategy here since it is possible to call
    125     # predict on a model without compiling it.

.../tensorflow/python/keras/engine/network.py in __init__(self, *args, **kwargs)
    137         'inputs' in kwargs and 'outputs' in kwargs):
    138       # Graph network
--&gt; 139       self._init_graph_network(*args, **kwargs)
    140     else:
    141       # Subclassed network

.../tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
    456     self._setattr_tracking = False  # pylint: disable=protected-access
    457     try:
--&gt; 458       result = method(self, *args, **kwargs)
    459     finally:
    460       self._setattr_tracking = previous_value  # pylint: disable=protected-access

.../tensorflow/python/keras/engine/network.py in _init_graph_network(self, inputs, outputs, name)
    284     # Keep track of the network's nodes and layers.
    285     nodes, nodes_by_depth, layers, layers_by_depth = _map_graph_network(
--&gt; 286         self.inputs, self.outputs)
    287     self._network_nodes = nodes
    288     self._nodes_by_depth = nodes_by_depth

.../tensorflow/python/keras/engine/network.py in _map_graph_network(inputs, outputs)
   1916     if all_names.count(name) != 1:
   1917       raise ValueError('The name &quot;' + name + '&quot; is used ' +
-&gt; 1918                        str(all_names.count(name)) + ' times in the model. '
   1919                        'All layer names should be unique.')
   1920   return network_nodes, nodes_by_depth, layers, layers_by_depth

ValueError: The name &quot;add&quot; is used 2 times in the model. All layer names should be unique."><pre class="notranslate"><span class="pl-v">ValueError</span>                                <span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>)
<span class="pl-c1">&lt;</span><span class="pl-s1">ipython</span><span class="pl-c1">-</span><span class="pl-s1">input</span><span class="pl-c1">-</span><span class="pl-c1">1</span><span class="pl-c1">-</span><span class="pl-c1">07</span><span class="pl-s1">b5faf0201d</span><span class="pl-c1">&gt;</span> <span class="pl-s1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>
      <span class="pl-c1">4</span> <span class="pl-s1">add1</span> <span class="pl-c1">=</span> <span class="pl-s1">inputs</span> <span class="pl-c1">+</span> <span class="pl-s1">inputs</span>
      <span class="pl-c1">5</span> <span class="pl-s1">add2</span> <span class="pl-c1">=</span> <span class="pl-s1">keras</span>.<span class="pl-s1">layers</span>.<span class="pl-v">Add</span>()([<span class="pl-s1">inputs</span>, <span class="pl-s1">inputs</span>])
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">6</span> <span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-s1">keras</span>.<span class="pl-s1">models</span>.<span class="pl-v">Model</span>(<span class="pl-s1">inputs</span><span class="pl-c1">=</span>[<span class="pl-s1">inputs</span>], <span class="pl-s1">outputs</span><span class="pl-c1">=</span>[<span class="pl-s1">add1</span>, <span class="pl-s1">add2</span>])

...<span class="pl-c1">/</span><span class="pl-s1">tensorflow</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">keras</span><span class="pl-c1">/</span><span class="pl-s1">engine</span><span class="pl-c1">/</span><span class="pl-s1">training</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">121</span>
    <span class="pl-c1">122</span>   <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">123</span>     <span class="pl-en">super</span>(<span class="pl-v">Model</span>, <span class="pl-s1">self</span>).<span class="pl-en">__init__</span>(<span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">124</span>     <span class="pl-c"># initializing _distribution_strategy here since it is possible to call</span>
    <span class="pl-c1">125</span>     <span class="pl-c"># predict on a model without compiling it.</span>

...<span class="pl-c1">/</span><span class="pl-s1">tensorflow</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">keras</span><span class="pl-c1">/</span><span class="pl-s1">engine</span><span class="pl-c1">/</span><span class="pl-s1">network</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">137</span>         <span class="pl-s">'inputs'</span> <span class="pl-c1">in</span> <span class="pl-s1">kwargs</span> <span class="pl-c1">and</span> <span class="pl-s">'outputs'</span> <span class="pl-c1">in</span> <span class="pl-s1">kwargs</span>):
    <span class="pl-c1">138</span>       <span class="pl-c"># Graph network</span>
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">139</span>       <span class="pl-s1">self</span>.<span class="pl-en">_init_graph_network</span>(<span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">140</span>     <span class="pl-s1">else</span>:
    <span class="pl-c1">141</span>       <span class="pl-c"># Subclassed network</span>

...<span class="pl-c1">/</span><span class="pl-s1">tensorflow</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">training</span><span class="pl-c1">/</span><span class="pl-s1">tracking</span><span class="pl-c1">/</span><span class="pl-s1">base</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">_method_wrapper</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">456</span>     <span class="pl-s1">self</span>.<span class="pl-s1">_setattr_tracking</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>  <span class="pl-c"># pylint: disable=protected-access</span>
    <span class="pl-c1">457</span>     <span class="pl-k">try</span>:
<span class="pl-c1">-</span><span class="pl-c1">-&gt;</span> <span class="pl-c1">458</span>       <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-en">method</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">459</span>     <span class="pl-k">finally</span>:
    <span class="pl-c1">460</span>       <span class="pl-s1">self</span>.<span class="pl-s1">_setattr_tracking</span> <span class="pl-c1">=</span> <span class="pl-s1">previous_value</span>  <span class="pl-c"># pylint: disable=protected-access</span>

...<span class="pl-c1">/</span><span class="pl-s1">tensorflow</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">keras</span><span class="pl-c1">/</span><span class="pl-s1">engine</span><span class="pl-c1">/</span><span class="pl-s1">network</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">_init_graph_network</span>(<span class="pl-s1">self</span>, <span class="pl-s1">inputs</span>, <span class="pl-s1">outputs</span>, <span class="pl-s1">name</span>)
    <span class="pl-c1">284</span>     <span class="pl-c"># Keep track of the network's nodes and layers.</span>
    <span class="pl-c1">285</span>     <span class="pl-s1">nodes</span>, <span class="pl-s1">nodes_by_depth</span>, <span class="pl-s1">layers</span>, <span class="pl-s1">layers_by_depth</span> <span class="pl-c1">=</span> <span class="pl-en">_map_graph_network</span>(
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">286</span>         <span class="pl-s1">self</span>.<span class="pl-s1">inputs</span>, <span class="pl-s1">self</span>.<span class="pl-s1">outputs</span>)
    <span class="pl-c1">287</span>     <span class="pl-s1">self</span>.<span class="pl-s1">_network_nodes</span> <span class="pl-c1">=</span> <span class="pl-s1">nodes</span>
    <span class="pl-c1">288</span>     <span class="pl-s1">self</span>.<span class="pl-s1">_nodes_by_depth</span> <span class="pl-c1">=</span> <span class="pl-s1">nodes_by_depth</span>

...<span class="pl-c1">/</span><span class="pl-s1">tensorflow</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">keras</span><span class="pl-c1">/</span><span class="pl-s1">engine</span><span class="pl-c1">/</span><span class="pl-s1">network</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">_map_graph_network</span>(<span class="pl-s1">inputs</span>, <span class="pl-s1">outputs</span>)
   <span class="pl-c1">1916</span>     <span class="pl-k">if</span> <span class="pl-s1">all_names</span>.<span class="pl-en">count</span>(<span class="pl-s1">name</span>) <span class="pl-c1">!=</span> <span class="pl-c1">1</span>:
   <span class="pl-c1">1917</span>       <span class="pl-s1">raise</span> <span class="pl-v">ValueError</span>(<span class="pl-s">'The name "'</span> <span class="pl-c1">+</span> <span class="pl-s1">name</span> <span class="pl-c1">+</span> <span class="pl-s">'" is used '</span> <span class="pl-c1">+</span>
<span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">1918</span>                        <span class="pl-en">str</span>(<span class="pl-s1">all_names</span>.<span class="pl-en">count</span>(<span class="pl-s1">name</span>)) <span class="pl-c1">+</span> <span class="pl-s">' times in the model. '</span>
   <span class="pl-c1">1919</span>                        <span class="pl-s">'All layer names should be unique.'</span>)
   <span class="pl-c1">1920</span>   <span class="pl-k">return</span> <span class="pl-s1">network_nodes</span>, <span class="pl-s1">nodes_by_depth</span>, <span class="pl-s1">layers</span>, <span class="pl-s1">layers_by_depth</span>

<span class="pl-v">ValueError</span>: <span class="pl-v">The</span> <span class="pl-s1">name</span> <span class="pl-s">"add"</span> <span class="pl-c1">is</span> <span class="pl-s1">used</span> <span class="pl-c1">2</span> <span class="pl-s1">times</span> <span class="pl-c1">in</span> <span class="pl-s1">the</span> <span class="pl-s1">model</span>. <span class="pl-v">All</span> <span class="pl-s1">layer</span> <span class="pl-s1">names</span> <span class="pl-s1">should</span> <span class="pl-s1">be</span> <span class="pl-s1">unique</span>.</pre></div>