Solved by colesbury in post #2 
              
              
                The manual seed to get matching random numbers passes on my local card and one particular V100 card that I could access. So I wonder if this behavior from pytorch is by design or a bug? 

You should get consistent random numbers if you’re using the same seed, PyTorch version, and CUDA version even …