This is what I plan to try, based on what we discuess yesterday. I just don’t have enough exprerience yet to know beforehand if it will work as I want it to.
One of the issues I’m trying to understand whether it will cause problems, is something you mentioned. By doing
dl_batched = torch.utils.data.DataLoader(ds_batched, batch_size=None, shuffle=True)
I will get the same batches, albeit in different order, every time. Will this be an issue? On the other hand, even Trainer()'s group_by_length() has to be doing something similar. I tried checking its implementation in the docs but I am not sure what it is doing to be honest.
I will eventually try the method you proposed to me here though, just to see if it works as expected!
EDIT: there is something in the dataloader docs called sampler, which “defines the strategy to draw samples from the dataset”. Perhaps creating something custom related to this might offer a more ‘expected’ solution to my problem.