I’ve read this thread closely and tried everything I can see on here as suggestions, and I still cannot get deterministic behavior during training. I’m using Pytorch 1.1.0 and Torchvision 0.3.0.
In my code I have the following:
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.manual_seed(1)
torch.cuda.manual_seed_all(1)
np.random.seed(1)
random.seed(1)

And my dataloader looks like this:
train_loader = torch.utils.data.DataLoader(
    train_dataset,
    batch_size=128,
    num_workers=0,
    shuffle=True,
    pin_memory=True,
    worker_init_fn=random.seed(1)
)

Is there a place I’ve missed setting the seed? Looking at the first few mini-batches of training, the differences in accuracy across starts is pretty stark.
Also, I don’t even get deterministic behavior when I disable RandomResizedCrop and RandomHorizontalFlip in my image transforms – so the non-deterministic behavior is happening somewhere else.
Any clues or pointers would be appreciated.