It sounds like you might want to try processing the images in both datasets first to equalize features like brightness etc.?

I think it will take some time, and I am unsure if it will enhance the final results. Probably I will merge them and calculate the mean and std to use the transforms.Normalize.