I have done more tests, and it seems like the original model is actually suffering from this too.
Just to summarize:


Here’s Arthur’s original tutorial 19.
I ran his TensorFlow code 26 a number of times and got success rate varying from 0.02% to 48%.
I ran my PyTorch code  27 and got similar distribution.

After doing some experiments and reading tutorials, I am still not sure what other approaches can be taken in regards to .backward function.
My current understanding of how the dynamic graph works in PyTorch is the following: when you create a Variable with requires_grad=True and perform some actions on it, once you reach some scalar Variable, for example, loss function, and call loss.backward, all requires_grad=True Variables will have their derivatives with respect to loss in .grad param. Is this correct?
Maybe someone can try to reimplement his tutorial showing the proper usage of PyTorch? I, personally, would greatly appreciate it.