Solved by SimonW in post #2 
              
              
                By default cudnn can use nondeterministic algorithms. Set torch.backends.cudnn.deterministic = True if you want deterministic behavior at the cost of efficiency.