UPDATE:
It seems after looking carefully at the outputs that the loss with the scope with torch.no_grad(): is actually the correct loss. And both MSELoss and L1Loss return a sum loss and not a mean loss. Essentially my outputs are of size (bsize,5000,3) and input the same size and when I take the L1Loss or the MSELoss I get something that if I divide by bsize x 5000 x 3 will be in the order of what I am seeing when I see the loss with the no_grad() scope.
This is a bigger problem and I am not sure how to solve it, I don’t really want to just divide by the size and dimensions of the input, that seems too hacky.