Would the output of the different Datasets and collate functions be the same or would they be different?
In the latter case I guess you don’t want to shuffle the indices/samples, since this would create “mixed” batches?
Maybe you could create a custom collate_fn, which uses different work flows based on the passed indices? I.e. would it work if you combine the two collate functions and change the behavior based on the currently loaded data?