Solved by ptrblck in post #2 
              
              
                Did you call to() on the model after the first epoch, e.g. in your validation loop? 
Could you post a code snippet to reproduce this issue?