I guess it depends on the distribution of the values for all images.
You could clip some “outliers”, i.e. very high (or low) values or normalize each image separately using its own min/max (if that doesn’t destroy the information e.g. as would be the case for depth images). What kind of images are you using?