I’m not sure I understand the issue completely.
Based on this statement:



 stadtgiraffe:

But if I set the model.train() inside the loop, the “overfitting” continues since I go back to train mode after evaluation. There is no peak in the loss function and my evaluation confidence continues to decrease (even below 0.01) even more while my training confidence converges for all three detection layers to 0.999.


it sounds like the expected result. I.e. the training and evaluation losses decrease properly in their corresponding mode or are you using model.train() for both, training and evaluation?