Sooo realise that available dataset classes only cover traditional problems in CV.
Most of the times you need to write your own generic dataset torch.utils.data.Dataset.
In fact you can write a dataset which contains two other datasets like:
import torch
from torch.utils.data import Dataset,TensorDataset

class MyDataset(Dataset):
    def __init__(self, *args,**kwargs):
        super(MyDataset, self).__init__()
        self.dataset_images = TensorDataset(*args,**kwargs)
        self.dataset_targets = TensorDataset(*args,**kwargs)
    def __len__(self):
        return None # Be creative here? largest of both... dunno
    def __getitem__(self, idx):
        return self.dataset_images[idx], self.dataset_targets[idx]
    # Bu here you can randomly sample both, the shorter, just code anything you need. 


Basically, you can code anything you want using any python code or libraries. You can randomly sample both subsets, you can make code to be deterministic and them random for the shorter… everything is up to you.
Inside init you define any tool you need and in getitem you carry out the workload (opening files and so)