title: Tensorflow: chaining tf.gather() produces IndexedSlices warning
body: <p>I'm running into an issue where chaining <code>tf.gather()</code> indexing produces the following warning:</p>
  
  <pre><code>/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients.py:90: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.           
    "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
  </code></pre>
  
  <p>The scenario arises when one layer indexes into the input layer, performs some operation on the corresponding slice, and then the next layer indexes into the result. Here's a representative example:</p>
  
  <pre><code>import tensorflow as tf
  
  ## 10-Dimensional data will be fed to the model
  X = tf.placeholder( tf.float32, [10, None] )
  
  ## W works with the first 3 features of a sample
  W = tf.Variable( tf.ones( [5, 3] ) )
  Xi = tf.gather( X, [0,1,2] )
  mm = tf.matmul( W, Xi )
  
  ## Indexing into the result produces a warning during backprop
  h = tf.gather( mm, [0,1] )
  ...
  train_step = tf.train.AdamOptimizer(1e-4).minimize( loss )
  </code></pre>
  
  <p>The warning arises upon definition of <code>train_step</code> and goes away if the second <code>tf.gather()</code> call is taken away. The warning also goes away if <code>X</code> is provided with an explicit number of samples (e.g., <code>[10, 1000]</code>).</p>
  
  <p>Thoughts?</p>