title: Why does this tensorflow loop require so much memory?
body: <p>I have a contrived version of a complicated network:</p>
  
  <pre><code>import tensorflow as tf
  
  a = tf.ones([1000])
  b = tf.ones([1000])
  
  for i in range(int(1e6)):
      a = a * b
  </code></pre>
  
  <p>My intuition is that this should require very little memory. Just the space for the initial array allocation and a string of commands that utilizes the nodes and overwrites the memory stored in tensor 'a' at each step. But memory usage grows quite rapidly. </p>
  
  <p>What is going on here, and how can I decrease memory usage when I compute a tensor and overwrite it a bunch of times?</p>
  
  <p>Edit:</p>
  
  <p>Thanks to Yaroslav's suggestions the solution turned out to be using a while_loop to minimize the number of nodes on the graph. This works great and is much faster, requires far less memory, and is all contained in-graph.</p>
  
  <pre><code>import tensorflow as tf
  
  a = tf.ones([1000])
  b = tf.ones([1000])
  
  cond = lambda _i, _1, _2: tf.less(_i, int(1e6))
  body = lambda _i, _a, _b: [tf.add(_i, 1), _a * _b, _b]
  
  i = tf.constant(0)
  output = tf.while_loop(cond, body, [i, a, b])
  
  with tf.Session() as sess:
      result = sess.run(output)
      print(result)
  </code></pre>