Why do we normalize only the X data and not Y I saw places saying that when using a NN you should SOMETIMES normalize your data: we take the X_train, subtract the mean and divide by the std.
Is the following statements correct, and if so, why?

*

*We only normalize the X data, and the y data stays as is.

Note: The similar Question associated with my post does not answer my question, it talks about why we normalize X, but not about the normalization of y and the reasons for it.

I don't agree. It may not make much difference if the values are small, but target variable directly affects the gradient size and can result in wild updates during back-propagation. Also, the behavior under regularization may dramatically change. It'll be harder to increase the weights to compensate for the increase in target variable.
On the other hand, note that not all problems demand for mean/std standardization. The target variable might be a probability, and the output activation might be a sigmoid. The above paragraph is intended for general unconstrained regression problems.
