Relative performance depends both on tensor shapes and hardware/software versions… One thing to try is batch_first=False mode. If that doesn’t help, then yes, indexing is a valid approach (at least for GRUs, with LSTMs hidden/cell state distinction complicates things).
Some random snippet of indexing (table is (time,batch,*) , gather_pts is a 1d tensor (last indices)):
brange = torch.arange(batch_size, dtype=torch.int64, device=table.device)
y = table[gather_pts, brange]

or you can do the same with torch.gather
You can also try GitHub - lmnt-com/haste: Haste: a fast, simple, and open RNN library, I think it outputs correct states without packing.