Thanks for your response @ptrblck, I overlooked “mixed batches”. This unified Dataloader would be used in BERT pre-training. The research paper suggests using one sequence length encoding first and then the second. Therefore, “To speed up pretraing in our experiments, we pre-train the model with a sequence length of 128 for 90% of the steps. Then, we train the rest 10% of the steps of sequence of 512 to learn the positional embeddings”.
It would be easier to define the custom collate_fn if the indices are passed to it but it only receives the samples/data directly, right? Perhaps, another approach now that the context is more clear?