I once had problems with model.eval because i was using batch normalization and couldnt support a large enough batch size. if you are using BN, could you try putting only those layers in eval mode while training and see if the train-val behave similarly?