The training loss decreases in training mode while the evaluation loss increases in eval mode (which is the basic sign for overfitting).
My issue is that I do not fully understand why training works better (much better mAP performance) if I freeze my batchnorm layer after epoch 1, meaning just continue training in model.eval() instead of model.train() after one epoch.
I tried adding Dropout additional to the BN layer to prevent fast overfitting but it still shows that my training works better if I switch to model.eval() for training.