<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04</li>
<li>TensorFlow installed from (source or binary): binary</li>
<li>TensorFlow version (use command below): 1.14 and 2</li>
<li>Python version: 3.6</li>
<li>CUDA/cuDNN version: 10/7</li>
<li>GPU model and memory: NA</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong></p>
<p dir="auto">Imagine you extend the tf.keras.layers.Layer with the class MyLayer, which is a wrapper for the tf.keras.layers.Dense layer class (this class extension could also be a Model, but that's not important).</p>
<p dir="auto">Now imagine you instantiate a Dense layer <code class="notranslate">dense</code> (w/ no bias) and initialize two independent MyLayer instances <code class="notranslate">dense1</code> and <code class="notranslate">dense2</code> using <code class="notranslate">dense</code>...</p>
<p dir="auto">Now, we extend the Model class to take two MyLayer instances on init. We do this because we want to use dense1 and dense2 to perform different tasks but we want their weights to be shared (in a realistic example, MyLayer would be split into MyLayer1 and MyLayer2, which would perform different operations on <code class="notranslate">call</code> but would both depend on <code class="notranslate">dense</code>). We run data through our model pipeline and compute a loss, gathering gradients by using either the GradientTape or keras optimizers. However, when we gather and apply gradients with respect to Model.trainable_variables, even though our model has one unique weight, the kernel for <code class="notranslate">dense</code>, Model.trainable_variables will return two variables, and the gradients will be calculated and thus applied twice, which I would say is a bug, insofar as it's unexpected.</p>
<p dir="auto"><strong>Describe the expected behavior</strong></p>
<p dir="auto">Models whose layers have shared/tied weights should not return duplicate weights when accessing the trainable_variables property. A super simple work-around is to call list(set(model.trainable_variables)) but the real issue is the unexpected behavior: "Why would I ever think that model.trainable_variables would return duplicates of the same variable ?!"</p>
<p dir="auto">The two calls to test in the code below should have the same output.</p>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
`<br>
code</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import numpy as np
import tensorflow as tf

tf.enable_eager_execution()

class MyDense(tf.keras.layers.Layer):
  def __init__(self, dense, **kwargs):
    super().__init__(**kwargs)
    self.dense = dense
  def call(self, inputs):
    return self.dense(inputs)

class MyModel(tf.keras.Model):
  def __init__(self, dense1, dense2, **kwargs):
    super().__init__(**kwargs)
    self.dense1 = dense1
    self.dense2 = dense2
  def call(self, inputs):
    return self.dense1(inputs) + self.dense2(inputs)

def test(unique):
  x = tf.ones(shape=(10, 5))
  y = tf.ones(shape=(10, 1)) + 2

  dense = tf.keras.layers.Dense(1, kernel_initializer=tf.keras.initializers.constant(0.2), use_bias=False)
  dense1 = MyDense(dense)
  dense2 = MyDense(dense)
  model = MyModel(dense1, dense2)

  adam = tf.keras.optimizers.Adam(0.001)

  with tf.GradientTape() as tape:
    y_hat = model(x)
    loss = tf.keras.losses.mse(y, y_hat)
    variables = model.trainable_variables if unique else list(set(model.trainable_variables))
    print(f&quot;# trainable variables: {len(variables)}&quot;)

    pre_weight = variables[0][0]

    grads = tape.gradient(loss, variables)
    adam.apply_gradients(list(zip(grads, variables)))

    post_weight = variables[0][0]

    return post_weight - pre_weight

ex1 = test(True)
print(ex1)
print(&quot;\n+=+=+=+=\n&quot;)
ex2 = test(False)
print(ex2)"><pre class="notranslate"><code class="notranslate">import numpy as np
import tensorflow as tf

tf.enable_eager_execution()

class MyDense(tf.keras.layers.Layer):
  def __init__(self, dense, **kwargs):
    super().__init__(**kwargs)
    self.dense = dense
  def call(self, inputs):
    return self.dense(inputs)

class MyModel(tf.keras.Model):
  def __init__(self, dense1, dense2, **kwargs):
    super().__init__(**kwargs)
    self.dense1 = dense1
    self.dense2 = dense2
  def call(self, inputs):
    return self.dense1(inputs) + self.dense2(inputs)

def test(unique):
  x = tf.ones(shape=(10, 5))
  y = tf.ones(shape=(10, 1)) + 2

  dense = tf.keras.layers.Dense(1, kernel_initializer=tf.keras.initializers.constant(0.2), use_bias=False)
  dense1 = MyDense(dense)
  dense2 = MyDense(dense)
  model = MyModel(dense1, dense2)

  adam = tf.keras.optimizers.Adam(0.001)

  with tf.GradientTape() as tape:
    y_hat = model(x)
    loss = tf.keras.losses.mse(y, y_hat)
    variables = model.trainable_variables if unique else list(set(model.trainable_variables))
    print(f"# trainable variables: {len(variables)}")

    pre_weight = variables[0][0]

    grads = tape.gradient(loss, variables)
    adam.apply_gradients(list(zip(grads, variables)))

    post_weight = variables[0][0]

    return post_weight - pre_weight

ex1 = test(True)
print(ex1)
print("\n+=+=+=+=\n")
ex2 = test(False)
print(ex2)
</code></pre></div>
<p dir="auto">`</p>
<p dir="auto">This is a little difficult to explain, and I'm guessing someone has to have run into it before, but it would be nice to at least get an explanation for what's going on here.</p>
<p dir="auto">The issue I'm finding is that when you specify a <code class="notranslate">model_dir</code> argument for any tflearn model, the first time you fit that model, the tensorboard warning like the following will always appear:</p>
<p dir="auto"><code class="notranslate">WARNING:tensorflow:Found more than one graph event per run. Overwriting the graph with the newest event.</code></p>
<p dir="auto">If you clear the contents of that directory and rerun the same model, the warning no longer appears.</p>
<p dir="auto">For example, here is a piece of code to reproduce:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
import random

from tensorflow.contrib.learn.python import learn
from tensorflow.contrib.learn.python.learn import datasets
from tensorflow.contrib.learn.python.learn.estimators._sklearn import accuracy_score
from tensorflow.contrib.learn.python.learn.estimators._sklearn import train_test_split
random.seed(42)

iris = datasets.load_iris()
x_train, x_test, y_train, y_test = train_test_split(iris.data,
                                                    iris.target,
                                                    test_size=0.2,
                                                    random_state=42)

def custom_optimizer(learning_rate):
    return tf.train.MomentumOptimizer(learning_rate, 0.9)

classifier = learn.TensorFlowDNNClassifier(
    hidden_units=[10, 20, 10],
    n_classes=3,
    steps=400,
    learning_rate=0.01,
    optimizer=custom_optimizer,
    model_dir='/tmp/tf/bug_test_1')
classifier.fit(x_train, y_train)"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
import random

from tensorflow.contrib.learn.python import learn
from tensorflow.contrib.learn.python.learn import datasets
from tensorflow.contrib.learn.python.learn.estimators._sklearn import accuracy_score
from tensorflow.contrib.learn.python.learn.estimators._sklearn import train_test_split
random.seed(42)

iris = datasets.load_iris()
x_train, x_test, y_train, y_test = train_test_split(iris.data,
                                                    iris.target,
                                                    test_size=0.2,
                                                    random_state=42)

def custom_optimizer(learning_rate):
    return tf.train.MomentumOptimizer(learning_rate, 0.9)

classifier = learn.TensorFlowDNNClassifier(
    hidden_units=[10, 20, 10],
    n_classes=3,
    steps=400,
    learning_rate=0.01,
    optimizer=custom_optimizer,
    model_dir='/tmp/tf/bug_test_1')
classifier.fit(x_train, y_train)
</code></pre></div>
<p dir="auto">If you run this in a jupyter notebook, wait until it is done, cd into <code class="notranslate">/tmp/tf/bug_test_1</code>, and run <code class="notranslate">tensorboard --logdir=.</code>, the warning will appear.  If you clear the contents and run again, it will no longer appear.  If you, in the same notebook within the same kernel, change the directory to something else (e.g. <code class="notranslate">/tmp/tf/bug_test_2</code>) and run again, the warning will again appear.  And the warning will again go away if you clear the contents and run again with the same directory.</p>
<p dir="auto">I also tried pre-creating the directory but that appears to make no difference.</p>
<p dir="auto">I wouldn't imagine this indicates a major problem but I try to take those warnings seriously since they have indicated more problematic issues in the past with running multiple models in the same kernel, so it would be great to know what this means.</p>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System: OS X Yosemite (10.10.5)<br>
Tensorflow Version: 0.10.0</p>