I was pointed out that I should model.eval() and model.train(), every time I want to call my models. It’s because I use batch normalization and it operates differently depending on train flag. Now the model learns correctly.
Is it a good idea to always wrap model calls with eval/train?
model.eval()
r = model(input)
model.train()