Are you lazily loading the data in all “small” datasets or are you preloading the complete data?
In the former case, could you check, if a specific Dataset, which is added to ConcatDataset, creates the slowdown?