Neural Networks, Gradient Descent.
affects your network learning can be seen in my notebook where I faced this problem with a slightly complex network (than the toy network above) and had to use Adam optimizer for gradient descent (along with Xavier/Glorot initialization of weights) so that I get a good set of gradients and gradient