Fair comparison of machine learning methods with different parameterization Let us assume the following scenario: The goal is to forecast the additional number of customers of a set of companies. To do this, you use a basic set of characteristics (e.g., company-specific characteristics and macroeconomic covariates) and are able to predict the figures quite well.
An employee at your company now says that she has collected a new dataset (e.g., 50 new features) that might be interesting to consider in the estimation process to further improve the performance of your machine learning models.
When you take the additional data into account, you notice better predictive performance, e.g. in terms of RMSE. However, the parameterization is also different because you have an additional set of 50 features.
In practice, can we say that the additional features make it possible to increase the performance metrics simply by comparing the two performance metrics? (Is this a fair comparison?) Or is it necessary to account for the different parameterization and the additional number of features?

If the only thing you care about is prediction (and not estimating impact size of input factors), then what you put in the model is not that important as long as it 'works'. However, adding more features increases chances of some of them being correlated with outcome by chance, so you'll need to carefully cross-validate and ideally compare not only mean estimates for R2 RMSE etc, but their confidence intervals. I.e. if you improve R2 from 20% +/-3% to 25%+/-10% this may not really be an improvement and the model will be overfit and less stable on new data. It may be easier to  bootstrap to find confidence intervals for all the metrics.
