Solved by ptrblck in post #3 
              
              
                Could you try setting torch.backends.cudnn.deterministic = True and run it again? 
cuDNN might use non-deterministic methods for a larger batch size, bun I’m unsure if that’s the case.