Or if you give to tanh values that are large in absolute value.

Hmm this could suggest why too.

Note that if you initially have gradients and then it goes to 0, that means that your model is properly converging a minimum value of the loss function that you provided.

But sadly, it seems like my accuracy doesn’t improve, and there’s still a loss value. So I think it’s likely some form of local minima? I’m not sure if I can tell also if that the network has sufficient representational power.