Solved by SimonW in post #4 
              
              
                Do you have cuDNN installed and enabled? 
If so, could you check if your observation is same as the issue described here? https://github.com/pytorch/pytorch/issues/3665