Since your use case seems to be a bit more complicated I would recommend to implement a custom Dataset as explained in this tutorial 21.
This would allow you to implement the loading logic (and correspondence of the image folders) manually.