Could it work just to use padded sequences?
If I understand it correctly, first padding the sequences and use the corresponding output ( e.g. input is [1,2,0,0] and output is [0,1,2,2], I will use the second output “1” to calculate the loss), don’t need any other operations in rnn layer?