The problem is in the data augmentation transformations:
transforms.RandomCrop(32, padding=4),
transforms.RandomHorizontalFlip(),

After I removed the above code, it worked 100% deterministically…
I am using Pytorch 0.3.1
Does torchvision have a different seed from torch?