I seem to have achieved better results by eliminating the first 10 data points of the spectrogram. It looks like there is a clear difference in the spectrogram for the first data points which causes this problem when normalizing the picture.
I assume this is something with the difference between MP3 and WAV that anything to do with Spectrogram ?