Hi Thomas,
I just read it and saw this:

There are some PyTorch functions that use CUDA functions that can be a source of non-determinism. One class of such CUDA functions are atomic operations, in particular  atomicAdd , where the order of parallel additions to the same value is undetermined and, for floating-point variables, a source of variance in the result. PyTorch functions that use  atomicAdd  in the forward include  torch.Tensor.index_add_() ,  torch.Tensor.scatter_add_() , torch.bincount() .
A number of operations have backwards that use  atomicAdd , in particular  torch.nn.functional.embedding_bag() , torch.nn.functional.ctc_loss()  and many forms of pooling, padding, and sampling. There currently is no simple way of avoiding non-determinism in these functions.

Just to make sure, I wonder if this means that as long as my script used one of these operations, there would be no way to guarantee deterministic training? Not even with torch.backends.cudnn.deterministic=True?
Thanks.