What range are your targets using?
If the targets are “classification” targets, i.e. zeros and ones, I would expect the model to try to output very high activations to saturate the tanh and this match the target values.