I think this is not about the momentum. I have the same problem. when I call
model.eval()
model(input)
model.train()
model(input)
model.eval()
model(input)
model.train()
model(input)

every call of model(input) is almost the same if it is after model.train() , but differs with what follows model.eval().