Image mode changes how we normalize the input image data. If the model is trained with some specific normalization then we should try to use that same normalization to match to achieve the best accuracy.