If you remove the model.eval() for testing purposes does it improve the results? It might be useful to print or track some tensors layer by layer to see where the divergence begins during training and validation.