title: Understanding tensorboard: why 12 tensors sent to optimizer?
body: <p>So I made the simplest model I could (a perceptron/autoencoder) which (aside from input generation) is the following:</p>
  
  <pre><code>N = 64 * 64 * 3
  
  def main():
      x = tf.placeholder(tf.float32, shape=(None, 64, 64, 3), name="x")
  
      with tf.name_scope("perceptron"):
          W = tf.Variable(tf.random_normal([N, N], stddev=1), name="W")
          b = tf.Variable(tf.random_normal([], stddev=1), name="b")
          y = tf.add(tf.matmul( tf.reshape(x, [-1,N]), W), b, name="y")
          act = tf.nn.sigmoid(y, name="sigmoid")
          yhat = tf.reshape(act, [-1, 64, 64, 3], name="yhat")
  
      with tf.name_scope("mse"):
          sq_error = tf.reduce_mean(np.square(x - yhat), axis=1)
          cost = tf.reduce_mean( sq_error, name="cost" )
          tf.summary.scalar("cost", cost)
  
      with tf.name_scope("conv_opt"): #Should just be called 'opt' here
          training_op = tf.train.AdamOptimizer(0.005).minimize(cost, name="train_op")
  
      with tf.device("/gpu:0"):
          config = tf.ConfigProto(allow_soft_placement=True)
          config.gpu_options.allow_growth = True
          sess = tf.Session(config=config)
          sess.run(tf.global_variables_initializer())
  
          logdir = "log_directory"
          if os.path.exists(logdir):
              shutil.rmtree(logdir)
          os.makedirs(logdir)
  
          input_gen = input.input_generator_factory(...)
          input_gen.initialize((64,64,3), 512)
  
          merged = tf.summary.merge_all()
          train_writer = tf.summary.FileWriter(logdir, sess.graph)
  
          for i in range(10):
              batch = input_gen.next_train_batch()
              summary,_ = sess.run([merged, training_op], feed_dict={x : batch})
              train_writer.add_summary(summary, i)
              print("Iteration %d completed" % (i))
  
  if __name__ == "__main__":
      main()
  </code></pre>
  
  <p>This produces the following <a href="https://i.stack.imgur.com/4NCf4.png" rel="nofollow noreferrer">tensorboard graph</a>.  Anyway I presume the thick arrow from 'perception' to 'conv_opt' (which should probably just be called 'opt', sorry) corresponds to the back-propagating error signal, (whereas ?x64x64x3 arrows correspond to inference).  But why <em>12</em> tensors?  I don't see where that number comes from.  I would have expected fewer, corresponding to really just to <code>W</code>, and <code>b</code>.  Can someone please explain what's going on?</p>