Are you shuffling the Datasets? If so, try to set shuffle=False and run it again.
However, I’m a bit skeptical if your approach will always work, as you might want to use multiprocessing in your DataLoaders, which might lead to inconsistent racing between the workers.
I would rather create a new Dataset, initializing both of your datasets in it and just return deterministically two samples in __getitem__.
The code would look like this:

class MyDataset(Dataset):
    def __init__(self, dataset1, dataset2):
        self.dataset1 = dataset1 # datasets should be sorted!
        self.dataset2 = dataset2

    def __getitem__(self, index):
        x1 = self.dataset1[index]
        x2 = self.dataset2[index]

        return x1, x2

    def __len__(self):
        return len(self.dataset1) # assuming both datasets have same length

dataset = MyDataset(dataset1, dataset2)
loader = DataLoader(dataset, batch_size=...)

Both your datasets created from ImageFolder should return sorted images.
If that’s not the case, you would have to pass the image folder paths to MyDataset, sort the image paths for both folders, and lazily load the images in __getitem__.