To what extent does the quality of data play in the accuracy of a model? I’ve worked with businesses where I am provided with a data set that I know is inadequate in terms of its signal with respect to the measure in which I am trying to predict. No matter the extent to which I perform data engineering to create new variables, or the extent to which I experiment with new models or new model parameters, the performance metrics of the models I use will always approach a ceiling point due to the limit of the nature of the variables that is being used. Hence the question I pose: whether one should attempt to perform feature selection, model selection, hyper parameter optimisation IF there exist this constraint?


the performance metrics of the models I use will always approach a ceiling point due to the limit of the nature of the variables that is being used. Hence the question I pose: whether one should attempt to perform feature selection, model selection, hyper parameter optimisation IF there exist this constraint?

Optimization might still be necessary. Even when the data does not allow a model to measure values accurately, it will always remain that some models are better than others.
More interesting is the question what the nature of the constraint is. Why do you have this constraint and is it truly some limit or do you not have enough data or are your models not detailed enough?
For example: Some variables are just very hard to predict. For instance, when I flip a coin or roll a dice then I might be able to predict the average outcome, but for a given single instance it is extremely hard to predict the outcome. This is when nature, whose basic laws are deterministic, appears random to us.
