I assume you are using ImageFolder, which won’t work in your use case, as it’ll create the targets for a multi-class classification, while you seem to be dealing with a segmentation use case.
If that’s the case, I would recommend to write a custom Dataset as described here 3.
In the __init__ you could define the image paths to the data and target images and load the corresponding pair in the __getitem__ method.