It is difficult to understand what your data organization is without knowing what the meaning of data1 and data2 are. Are they different classes? Different datasets? If they are different classes, then a DatasetFolder or ImageFolder is exactly the abstraction for this this scenario: torchvision.datasets — Torchvision 0.8.1 documentation (pytorch.org) 2