I see >5x slowdown on the .backward() call when using pack_padded_sequence (~80s instead of ~14s). Is this expected?
I am currently using a stack of 2 bidirectional lstms, what is the best practice for retrieving the final states when running on variable length inputs? I did this (not sure if correct / fastest approach):

rnn_out, (ht, ct) = lstm_layer(lstm_input) 
reshaped_hidden = ht.view(num_lstm_layers, 2, batch_size, hidden_size)
back_forward_concat = torch.cat([reshaped_hidden[-1, 0, :, :], reshaped_hidden[-1, 1, :, :]], dim=1)

Thanks!