Overestimating the lower values and underestimating the higher values in Regression I've tried many different models from Linear to non-linear and flexible models like random forest to solve a regression problem. But all of them apparently overestimate the lower values of the response variable and underestimate the higher values. Below you can see the "observed versus fitted" plot of the random forest model:

Question: What are the possible reasons behind this issue and how it be fixed?

[Edited as I misread the original question]
The way a Random Forest works, simplified, is that the computer tries to sort the observations into buckets, then calculates the average value within each bucket, and uses that as the score for everything in that bucket. This is done repeatedly, getting different bucketing rules, and then the final score for an observation is the average of the scores for all the buckets it was in.
One issue with Random Forest is that it can "top out": a set of observations can be sorted into the "highest bucket" (or the "lowest bucket"), and from there, no further distinction is made within the set. If the same set is topping out for each bucketing rule, then you're not going to be getting a different score for them.
Something along these lines appears to be happening with your predictions. Your model isn't predicting less than -1, or much more than 1, which suggests that the set all observations with a true value greater than 1 is topping out, and vice versa for less than -1.
Now, your data also seems to be much more sparse for those data points, so your model is probably prioritizing the more central ones. Furthermore, you may not have enough datapoints, regardless of what you do, to make predictions for those observations. Or the explanatory variable may just not be explanatory anymore at the extremes (for instance, age is a good predictor of height for schoolchildren, but not so much for adults; if you have that sort of relationship, there might not be anything you can do). But things you can try are:

*

*Create separate models for the extremes

*Look at your extremes and try to find distinguishing characteristics

*Generate new features, such as doing a linear regression on the original features, and feeding the output of that into the Random Forest model as an input.

