Solved by ptrblck in post #2 
              
              
                I would recommend to check out the performance guide, which would point to towards using e.g. torch.backends.cudnn.benchmark = True and to e.g. avoid synchronizations via item() calls.