It depends on if the datasets are expected to produce the same kind of labels/annotations (e.g., are you doing classification or object detection)? If you are just doing classification I guess you can just use an ImageFolder provided the organization of the dataset is classes separated by folders.