Solved by ptrblck in post #2 
              
              
                Which criterion have you used? 
It seems you used a criterion, which is not suitable for multi-target prediction, e.g. NLLLoss. 
Could you try it again with e.g. MSELoss?