Hello,
I’m training a model which seems like it’s learning well during the training step. Even though, during the validation step the performance is terrible, and seems like the model didn’t learn at all. The really low accuracy  on the validation set is perceptive at the second epoch and it keeps going lower and lower during the training, while the training accuracy keeps going up normally.
I’m fine tuning a MobileNetV3_small from torchvision models and training it from scratch or pretrained just freezing its weights. Both approaches are having the same problem.
I’ve tried to remove the model.eval() and model.train() from my code, and the performance seems to be so much better.
I don’t know why it’s happening. The training and validation sets seem to be working well.