title: Tensorflow: How to Propagate Gradient Through tf.gather?
body: <p>I'm having some issues trying to propagate the gradient of my loss function with respect to a variable that represents the gather index, similar to what is done in spatial transformer networks (<a href="https://github.com/tensorflow/models/blob/master/transformer/spatial_transformer.py" rel="noreferrer">https://github.com/tensorflow/models/blob/master/transformer/spatial_transformer.py</a>). I feel like I might be missing something very simple. Here is a simplified toy example of what I would want to do:</p>
  
  <pre><code>import tensorflow as tf
  import numpy as np
  lf = np.array([1.0,2.0,3.0])
  lf_b = 2.0
  sess = tf.InteractiveSession()
  x = tf.placeholder(tf.float32, shape=(3))
  pt = tf.Variable(0, name='point')
  y_ = tf.placeholder(tf.float32, shape=())
  sess.run(tf.initialize_all_variables())
  y = tf.gather(x, pt)
  data_loss = tf.reduce_mean(tf.squared_difference(y,y_))
  train_step = tf.train.GradientDescentOptimizer(0.01).minimize(data_loss)
  </code></pre>
  
  <p>Currently, this returns the error: 
  ValueError: No gradients provided for any variable</p>