Maybe I wasn’t precise enough. It’s not that my accuracy slowly drops to 0, or that I’m wrongly training.
Because in that case, it could be a problem with the optimizer state dict etc. (I checked that possibility)
The problem is that, even without training, without doing any loss.backward() or optimizer.step(), I already have a loss that indicates that my model is garbage when it’s configured with model.train(). But everything is ok when I use model.eval(). For the exact same code, dataloader etc…
It’s like if the train() method was making my model completely useless.
It’s not the training procedure that I don’t even have to execute to already see the big difference in loss between train() and eval(). I’m doing fp32 training. I know that with misconfigured fp16 training you could have bad eval() and good train(), but here it’s the opposite.
Is there a way to reconfigure the model.train() the same way it’s configured when I do model.eval()? like manually re-modifying the batchnorm layers etc… ?