Solved by ptrblck in post #2 
              
              
                You could write a new custom Dataset, pass both datasets to it and use your logic inside __getitem__ to draw the corresponding labels from your ImageFolder and datasetA.