That should generally be sufficient. A few operations have non-deterministic gradient calculations. For example, the gradient calculation of index_select() with duplicate indices can result in non-determinism due to the ordering of floating point additions.
The description of the operators with non-deterministic behavior is here:
https://pytorch.org/docs/stable/notes/randomness.html#pytorch 245
Try to narrow down where the non-determinism is introduced. Use backward hooks 37 to print out where in the network the gradient starts to vary between runs.