Your issue might be related to this discussion 71.
Basically, even if you set torch.backends.cudnn.deterministic = True, torch.backends.cudnn.benchmark = True might select different deterministic algorithms, which would yield different results.
There is a current PR 25 working on this.
Could you disable benchmarking and try it again?