I think the proper approach would be to write a custom Dataset (e.g. as given in this tutorial 7) and load the images from both folders simultaneously. This would make sure to get the desired image pairs in the same order without trying to use the seeds etc.
Let me know, if this approach would work.