Widespread overfitting in health domain research? I was reading about flaws with model selection techniques such as elimination based on significance and backwards selection via AIC (or similar) in the context of regression leading to inflated coefficients, narrow confidence intervals, and p-values that are lower than they should be.
In the health domain, the area in which I work, it's not uncommon to see such techniques used in journal articles. Whilst techniques such as train/test splitting and cross-validation can reduce this issue, these are also often not used since the primary goal is statistical inference and not prediction per se.
This led me to think about the following questions:

*

*Because much research in the health domain uses such model selection techniques without correcting for them does this mean it's likely that many models in this domain are overfitted, and therefore the results of these articles are inflated?


*Even though the primary goal is inference and not prediction, it still seems to be these regression models can be overfitted. Since these articles often do not train:test or cross-validate, is this a further source of overfitting that could be very widespread in this domain?


*Although machine learning is mostly concerned with prediction, could it not also be a valid option for describing relationships whilst also reducing this proposed overfitting that I hypothesise is occurring when researchers use such statistical methods? After all, if an algorithm can predict with decent performance on an unseen dataset then it clearly "understands" the relationship between the variables and the output (whilst some algorithms may be black-box, many have feature importance tools (RF, XGBoost) and xAI techniques can be considered to elucidate these relationships).


*Since machine learning techniques are often better equipped when features are many, could an appropriate solution to model selection be deriving the most important features from machine learning algorithms that perform best and then using those for prediction. This seems like a better choice than model selection based on significance or backwards selection.
I am an expert in none of the fields related to this question so apologies for my ignorance.

You are correct that overfitting is a rampant problem in health research, just as it is in all other fields in which sample sizes are not huge.  One of the biggest mistakes being made in recent years is to assume that machine learning algorithms somehow fix this problem.  While algorithms can be tuned with cross-validation to not overfit, many such as random forests typically result in massive overfitting.
It is a not correct to use one method of supervised learning to select features to promote for use in another method.  The second method has lost the context and does not know how to apply the proper amount of shrinkage.  In addition, the first method has a very low chance of finding the "right" features.  For example many practitioners think that lasso finds the right features when it fact it usually fails miserably in that task.
I go into many of these issues at length in RMS and BBR.
The most general-purpose, safest, and interpretable solution is heavy use of unsupervised learning (sparse principal components; regular principal components after doing variable clustering, etc.).  This allows either traditional or ML supervised learning to be used on the reduced, combined, features with much stability and without much overfitting.
Data are not capable of informing us about which variables are important, so we should stop trying to use data to do that.  A simple simulation in RMS shows that.  With a limited number of candidate features, a high signal:noise ratio, and zero collinearity, stepwise variable selection still has a very low chance of selecting the right variables.  Same with methods such as lasso.  If it can't work in an ideal setting it can't work on real data.
