Most likely some of your (validation) images are grayscale images, thus they only use a single channel.
You could check the shape of each loaded image and repeat the channel dimension to create an RGB image (which would of course still be gray, but would contain the necessary 3 input channels).