Hi,
I’m trying to train a network using cudnn but at every execution I’m getting different results. I have no idea why, as I’m trying to ensure determinism in every way I know.
Here is what I’m currently using to do so:
torch.manual_seed(0)
numpy.random.seed(0)
random.seed(0)
torch.backends.cudnn.deterministic = True
also I use num_workers=0 in the dataloader and I have manually checked that the input data on the network is always the same in every execution.
The parameters of the network also are also initialized in the same way, but as soon as the second/thirds batch comes in, some parameters and outputs of the network start to change sligthly leading to diferent training results.
Am I missing something?
Thanks.