I would recommend to write a custom Dataset class as described here 84, load the image pairs in the __getitem__ method and concatenate them there as well.
I assume you have some mapping between the RGB and grayscale images.
If so, you could pass the mapping together with the paths for the images to __init__ and load the pairs lazily in __getitem__.