The Problem
Even after setting all random factors with seeds, PyTorch result changes on the exact same setting if run sequentially.
To Reproduce
I have created 2 Kaggle Kernel that trains & test PyTorch on the same MNIST dataset five times. The kernels showcase the difference between setting random factors only once vs. each sequential run.


Setting random factor once 4


Setting random factor on each sequential run 2


Expected behavior
I expect running the same train & test five times on kernel #1 4 would produce the exact same result each time. But somehow, something changes on each iteration. I can’t pinpoint what changes on each iteration. Only by resetting the random factor on each iteration (kernel #2 2) I could achieve the exact same result each time.
Environment
Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: None

OS: Debian GNU/Linux 9 (stretch)
GCC version: (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
CMake version: version 3.7.2

Python version: 3.6
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip] msgpack-numpy==0.4.4.3
[pip] numpy==1.18.1
[pip] numpydoc==0.9.2
[pip] pytorch-ignite==0.3.0
[pip] pytorch-pretrained-bert==0.6.2
[pip] pytorch-transformers==1.1.0
[pip] torch==1.4.0
[pip] torchaudio==0.4.0a0+719bcc7
[pip] torchtext==0.5.0
[pip] torchvision==0.5.0
[conda] blas                      1.0                         mkl  
[conda] cpuonly                   1.0                           0    pytorch
[conda] mkl                       2019.3                      199  
[conda] mkl-service               2.0.2            py36h7b6447c_0  
[conda] mkl_fft                   1.0.12           py36ha843d7b_0  
[conda] mkl_random                1.0.2            py36hd81dba3_0  
[conda] pytorch                   1.4.0               py3.6_cpu_0  [cpuonly]  pytorch
[conda] pytorch-ignite            0.3.0                    pypi_0    pypi
[conda] pytorch-pretrained-bert   0.6.2                    pypi_0    pypi
[conda] pytorch-transformers      1.1.0                    pypi_0    pypi
[conda] torchaudio                0.4.0                      py36    pytorch
[conda] torchtext                 0.5.0                    pypi_0    pypi
[conda] torchvision               0.5.0                  py36_cpu  [cpuonly]  pytorch