I am experiencing some inconsistent predictions when running the same file multiple times. I followed the https://pytorch.org/docs/stable/notes/randomness.html 2 guide to set all the seeds and enable/disable the respective cudnn flags.
Training a small 3D network with some pooling layers on the CPU runs smoothly and the results between multiple runs are consistent. When training on the GPU the network predictions change after a few iterations even though the exact same network (and initialization) is used. Excluding the pooling layers fixes the problem. Furthermore, I can not observe such inconsistencies when using an equivalent 2D model (with pooling layers).
Below is a generic 3D examples which replicates my observations on my system.
Note: I’m not referring to differences between the CPU and the GPU run, I’m referring to multiple runs on the CPU/GPU respectively.

  
      gist.github.com
  
  
    https://gist.github.com/mibaumgartner/59f0be2e0a667bb9c1bec1106e49989e 9
Example
set_seed = 0
DEVICE = 'cuda'

import torch
torch.manual_seed(set_seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
import torch.nn as nn

import random
This file has been truncated. show original




  
  
    
    
  
  


Tested on:
Ubuntu 18.04
Tried with: Torch 1.0post2 with cuda 10
Torch 0.4.1 with cuda 9
Nvidia GTX 980
Edit: updated description and replaced 3d resnet examples with much simpler/smaller network.