Ah, I figured it out.
Basically you can floor the float index, then interpolate the int indices using the float one. That’s autograd friendly.

  
    
    
    Indexing a variable with a variable 
  
  
    Your tensorflow link is to a private repository so I can’t view it. 
The index operation doesn’t have any gradients defined w.r.t. the index variable. That’s not a limitation of “define by run”, that’s a property of the operation: it has integer domain. 
You need a differentiable sampling operation for spatial transformer networks. You can implement STN in PyTorch in roughly the same ways as in Tensorflow: sample floor(idx) and floor(idx) + 1 and linearly interpolate between the two. Note that t…