Hi @vdw and thanks for your help!
What would happen in such case if I were to do my initial hidden states learnable as suggested here 2? I am confused in the sense that now a whole bunch of time steps during training are on the same batch. I assume I should just copy the hidden states values to the new instance, but I am not sure if the ones to be copied in such case are the topmost (i.e. [:, 0, :]) bottommost or (i.e. [:, batch_size - 1 :]).