Yes, for now you have to constantly pack and unpack if you’re mixing RNNs and conv layers. If you’re mixing RNNs and fully connected layers, you don’t have to unpack at all – you can actually call the linear layer directly on the packed sequence.
The list just has to be sorted within an individual batch, so you can still shuffle your dataset and randomly sample a batch at a time, then sort each batch to send it to pack_padded_sequence.