Hello. It is only after training the network that I get the problem. I was able to get valid gradients by deleting the tanh application. It seems the problem was that after applying tanh the output would be pure 1s and -1s because of rounding up. So small perturbations of the input did not have any effect on the numeric value of the loss…
Could that be an explanation for the gradient appearing to be zero?