I know how to handle variable size sequence data in PyTorch for most cases. For example presume we want to predict next position of a person given its past locations. In this case we can pad different sequences with  pad_sequence  function. Then we can use  pack_padded_sequence  and  pad_packed_sequence  to calculate a batch of sequences of diverse lengths. Finally, We can use a mask on output of LSTM and ground truth positions and calculate loss and do back propagation.
In case of above example I’m aware of handling variable size sequences during training.
But assume another application of LSTM that we use hidden state of LSTM in last time step and concatenate it with other features and pass it to a deep neural network. Finally the deep neural network gives us a loss. I want to train all networks in an end to end fashion. But in this case I don’t know how to handle variable size sequences. In this case we don’t have ground truth for LSTM for different sequences, so we can’t use masking after  pack_padded_sequence  and  pad_packed_sequence . Instead we have a loss that back propagated from deep neural network.