Ok, thank you for that explanation… But then it seems that a STN isn’t possible under that condition. I know this is very broad, but is there any way to index or grab values from a variable tensor with another index-like variable that does have require_grad=True besides using detach()?  torch.gather doesn’t work either… Is scatter an option? It seems straight-forward in TF to so-called “differentiate the index”… is this just a limitation of define-by-run?
Here’s a fairly straight-forward STN gist 51 I made showing how it should work except for the error in the indexing. And here is 29 a tensorflow version of the transformer layer.
No worries though - if anyone else stumbles upon this and has insight, let me know 
EDIT: eh, I guess I see in the TF example above and the defomable conv pytorch 37 example how they still propagate the gradient through other means than the index… still unclear if differentiating the index is an inherent limitation of pytorch or in general.