Is "Exploratory Data Analysis" Fundamentally At Odds With "P-Hacking"? In our Statistics Class, we are learning about something called "P-Hacking" (also called "Data Snooping", "Data Dredging", etc.).
Here is the example we are working with in our class - we have a large dataset (e.g. over 5 million rows of data) that contain medical data on patients (each row is a medical patient). This includes information such as their age, gender, height, geographical location of residence, weight, income, highest obtained education level - and whether or not they have asthma. We are discussing strategies to perform analysis on this dataset - for example, do certain groups of people have asthma at higher rates compared to other groups of people.
Our professor told us we can consider different comparisons - for example:

*

*Do Men have asthma at higher rates than Women?

*Do people with University Degrees have asthma at higher rates compared to people without University Degrees?

*Do Men with University Degrees have asthma at higher rates compared to Women without University Degrees?

But as we see, the possible comparisons we can do with only the categorical variables are numerous - and we start factoring in the continuous variables (e.g. Men with University Degrees over the Age of 53 vs. Men with University Degrees over the Age of 53), the number of possible comparisons become even more.
Several of us in the class had this idea of "exploring the data" and seeing what interesting relationships we could find, and then trying to incorporate our findings into new hypothesis questions. But to our surprise, our professor heavily criticized this idea and called it a "classic example of P-Hacking". Our professor told us that for research to be taken seriously - all hypotheses (e.g. which comparisons) must be clearly stated prior to doing the research. Supposedly, this reduces the chances of "getting lucky" and finding some coincidental findings that are the product of chance. Our professor listed several examples of P-Hacking:

*

*Removing and inserting variables into a regression model arbitrarily until desirable p-values are found

*"Fishing" for different hypotheses until one of these hypotheses results in a desirable p-value

While these examples that the professor listed do seem relevant to me - I am not sure if I agree that these practices are "inherently" wrong. For example:

*

*In Machine Learning, there are entire algorithms on "Feature Selection" (e.g. Stepwise Selection, Genetic Algorithm, https://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html) that try different combination of features (and even create new features from old features) until a good model is produced. As I understand, these techniques can be used for a regression model. I understand that these methods can be prone to abuse with regards to P-Hacking (and result in overfitted models that are too complex to understand) - but if large datasets are available, couldn't procedures like Cross Validation be used to rigorously test these models to make sure that a "newly discovered insight" is not occurring by chance, but rather this new insight consistently appears across different levels of variability within the data?


*Regarding the second point, I also agree that "newly discovered trends, patterns and results" can be prone to abuse and can be misleading - but aren't "preconceived hypotheses" equally subject to such abuse and be have the potential to be equally misleading? Suppose I take some two random subset of medical patients (e.g. Men over 34 with University Degrees earning under 44k vs. Men under 34 with University Degrees earning more than 44k ) - I run some statistical comparison tests and find that "Men over 34 with University Degrees earning less than 44k" have asthma at very high rates. In the spirit of Machine Learning, could I not use some Cross Validation themed approach (such as Bootstrap), take repeated random samples of "Men over 34 with University Degrees earning less than 44k" (provided the sample size of this subgroup is large enough) and plot the distribution/histogram of their asthma rate? Could I also not use a similar approach like this to test the statistical significance between groups? Could such an approach not be used to partly determine if P-Hacking is occurring?


*The prof brought up that "if you fish for something long enough, you will eventually find it". By this, the prof meant that testing many different hypotheses will inevitably result in some false hypotheses being accepted and some true hypotheses being rejected. Doesn't the Bonferoni Correction Factor assist in adjusting the statistical significance so that the probability thresholds are more "strict" and result in hypotheses on the "cusp of being accepted" fall on the rejection side?


*And finally, if the entire research procedure, trials and errors, all hypotheses and results are formally stated (e.g. even the results that look "less attractive" with insignificant p-values) - couldn't this also be used to argue the case that P-Hacking and other malicious research practices did not occur?
Can someone please comment this - if my logic is correct, wouldn't this make aspects within Machine Learning and Exploratory Data Analysis fundamentally at odds with P-Hacking? Or can large datasets and Cross Validation be used to partly circumvent and mitigate such issues?

The problem with p-hacking in my view is not the hacking itself, but rather the interpretation of the outcome. Your professor is right saying that if you test and test and test, you will at some point find significance even if nothing is going on. This means that significances that you found in this way cannot be reliably interpreted as meaningful.
However, if you get at things from an exploratory angle, this should not be the point. Whatever turns up from an exploratory analysis should not be interpreted as a meaningful final result (some people argue that significances from testing should never be interpreted in this way, but I leave this discussion out here). Rather it should be interpreted as a potential hint at something, possibly be investigated on new independent data. If you look at certain data, test one hypothesis and get $p=0.44$ and another and get $p=0.0002$, the data that you have are more critical against the second null hypothesis than against the first, and there may be something going on. This kind of information is gradual and not dichotomous; If you run lots of tests, you should know that $p=0.04$ isn't particularly convincing, $p=0.012$ or $p=0.009$ may still occasionally happen accidentally, but $p=10^{-8}$ is still quite a strong indication (unless you have so much data that basically everything becomes significant; and also always look at effect sizes!).
Digression: There are formal procedures for multiple testing such as Bonferroni, Benjamini-Hochberg etc. I think it is good to know about these, because they give some information about what kind of p-value to expect in situations where multiple testing goes on. However, I think in EDA their worth is that they provide some orientation, but they shouldn't be literally followed and "trusted". They are based on simplifying assumptions (e.g., Bonferroni can be very conservative), and they can't take into account all the informal, visual snooping around that is done in EDA. Keep in mind that the "p-hacking" problem does not only come from running multiple tests, but also from running tests conditionally on informal decisions ("I run test $T$ because this connection here looks nonlinear, so I try out adding a squared term and test it"), which runs counter to the standard assumptions of the tests. (End digression.)
Ultimately I'd use p-values as exploratory indicators for things that may be worthwhile investigating without interpreting them as final meaningful result.
Another issue is the automatic use of multiple significance tests inside a procedure that ultimately does something else, for example a variable selection/prediction routine. You are right that the success of these procedures can be tested and validated on independent data, and the procedure can be seen as good to the extent that it leads to a model with good prediction power, if that's your ultimate aim. The issue here is the same as before: The significance tests are not interpreted as individually meaningful; i.e., you shouldn't claim that just because a model with variable $X_4$ was selected in such a procedure you have significant evidence that variable $X_4$ is important in a meaningful way. The tests here are not used as tests in the usual sense, but rather as building blocks of a method that actually does something else, and has to be assessed on its own merits.
Now of course one can ask whether it is a good idea to use significance tests in situations in which they cannot be interpreted in the standard way. My answer would be: It depends. You can hear often these days that for variable selection regularisation using ideas such as the Lasso is better than stepwise selection using significance tests, sometimes with the implicit suspicion that this is because the significance tests are used "wrongly". Well, my experience is, sometimes this is true and sometimes it isn't. I have seen data sets on which cross-validated prediction worked better using stepwise regression than the Lasso. If you don't have very many variables, this may not even be that exceptional. Also, when doing exploratory analysis, I like to look at p-values just to address the question as how "surprising" a find should actually be seen given a model in which nothing meaningful is going on behind it. However it is clear that this is just one bit of information, and the detailed visual impression and things such as effect sizes are still very relevant. Also of course I should not forget that "p-hacking" is going on, i.e., that the "model in which nothing meaningful is going on" is already violated to some extent by me snooping around. If I could formalise my "snooping" in advance, i.e., formulate a battery of things to test and maybe even some things to look at conditional on earlier results, I can generate data from a null model and in fact explore what kind of p-values I should expect to find in such a situation using the whole battery ($10^{-8}$ not often, I suspect).
A key to suitable interpretation may be to say goodbye to the idea that significance is something inherently good and meaningful. If you look at the data in order to find significances, you will find significances many of which will likely be meaningless. We should try to address the questions that are meaningful to us using the data, and then a (strong) significance is just something of a marker for further attention.
