Vigneswaran:

Now, does PyTorch automatically detect sequential processing in forward() and perform BPTT on its own when doing loss.backward() (i.e. without writing backward())?


No, nothing is “detected”, but backprop works for implementations without inplace operations (that usually accumulate outputs in a python list first). There was a blog post on this site about how JIT speeds up rnn unrolling, but c++ coded rnns are faster. You may check SRU 9 as an example that has python and c++ implementations.