Hi @tom thanks for your answer.

I don’t think it’s the algorithms as much as the lack of people doing it.
For example, for many operations (Upscaling comes up frequently) it would be relatively easy to make the cuda backward deterministic by having each  grad_in  location treated by one thread.

I see what you mean. It is not a problem of knowledge it is more a human resources problem, right? However, in my opinion, should be a priority to have a deterministic mode for all the current algorithms. I know that there is a general idea that non-deterministic algorithms do not have a lot of influence on the results, but, in some cases, there is a huge impact. For instance, I started this thread because in pytorch_geometric some of the operations are based on non-deterministic implementations of pytorch. As you can see in this github issue 11 this behavior is impacting the results. At the end this non-deterministic behavior has an impact in other libraries like pytorch geometric, causing the lack of reproducibility.

For others, using a  index_add / scatter_add -style operation, the  Embedding  backward may serve as a blueprint.

Can you give me more details about this? I am not fluent in cuda but if I have a blueprint I can try to collaborate to have a deterministic mode in statter_add/index_add. I think the deterministic mode is a must to have.