Can’t really tell unless you provide more details about the data, and even give some data points examples. (Labels included)
Are you normalizing the input pixels?
Meaning, does each channel from 0-255? Or 0-1? Consider divide by 255 to map to [0,1] if you didn’t do that.
Roy