Ideally that should be fine.
You seem to be using dataparallel and MSELoss inside your model definition. Is that right or Am I missing something?
What if you use MSE Loss once after you collect the output from the forward pass?