RMSE loss tends to output the same prediction When using MSE (or RMSE) loss for regression tasks in deep learning my models usually output the same result for each record. My guess is that the models are "playing safe" and they output a value towards the mean instead of learning anything.
Does anyone have encountered this problem or knows how to solve it?

My first thought was that this indicates the variables being unrelated to the outcome, in agreement with Stephan Kolassa.
However, if a large neural network cannot even overfit, that suggests to me some issue with the optimization. If your variables are unrelated to the outcome, you should expect poor out-of-sample performance, but you should be able to get something out of the in-sample data.
