Solved by ptrblck in post #2 
              
              
                RNNs expect an input in the shape [seq_len, batch_size, features] in the default setup. 
The temporal dimension is thus in dim0 and the first time step would be at index 0.