Have you looked into pad packed sequence and pack padded sequence? This blog post https://medium.com/huggingface/understanding-emotions-from-keras-to-pytorch-3ccb61d5a983 106 has a section describing packed sequences which might be what you are looking for unless I am misunderstanding your question.