I recently stumbled upon the same issue.
The same exact training and validation procedure, same parameters, same network, same data (as a sanity check) but still, the training loss was 3 orders of magnitude higher than the validation one, i.e. 50 during training and 0.05 in validation.
While trying to figure out what is happening I tried the following :

As mentioned above, removing torch.no_grad() during validation, did actually bring them to the same range, namely the large one.
Calculating the training error with a custom mse function instead of F.mse_loss(), brought them both to the smaller value.

What I am working on right now includes a 3-step training procedure and I am really not sure what causes that behavior as I have never encountered the same problem before the last step. The only difference between them is that the network for the first two does not have a recurrent layer and is not required to do a multi-step prediction by using its own predictions as next inputs recursively.
Could it be something related to the fact I am concatenating the recursive costs and then apply the mse function on that structure?