Yes, it would be the same if you don’t do detach and do retain_graph  .
In terms of the incorrect gradients, I was indeed talking about the detach option. Sorry for the confusion.
Are your training data very very long sequences? If they were multiple sequences, I’d still try bwd on full sequence and see which is faster, because cudNN is very fast. You can still evaluate every several sequences.