Could you try to play around with some hyperparameters (learning rate, weight init etc.)?
Your model without BatchNorm layers might be more sensitive to the input distribution.
Are you normalizing the image?