Do you use select and indexing somewhere and then change the tensor you indexed?
As those will share memory (they are 'view’s into the same memory in autograd lingo) they share the same version counter and modifying the “parent” tensor will invalidate using the view in the backward (as it doesn’t track which part was viewed and which part was modified, that is the case regardless of the actual values have changed).