For semantic segmentation I trained a network using a learning rate of 5*10**-4, mini-batchsize 2 and a Momentum of 0.8 with image size 256x384. The only custom transformation applied is a center crop. The loss after 20 Epochs looks like the following:

lr5e-4b2x256x384Mom8-noTransforms-reduceUpsample.png1280×960 54.2 KB

Question:
Why is there a peak in the loss function, when there is no custom transformation applied? Is the network to small?