Is "Exploratory Data Analysis" Fundamentally At Odds With "P-Hacking"? In our Statistics Class, we are learning about something called "P-Hacking" (also called "Data Snooping", "Data Dredging", etc.).
Here is the example we are working with in our class - we have a large dataset (e.g. over 5 million rows of data) that contain medical data on patients (each row is a medical patient). This includes information such as their age, gender, height, geographical location of residence, weight, income, highest obtained education level - and whether or not they have asthma. We are discussing strategies to perform analysis on this dataset - for example, do certain groups of people have asthma at higher rates compared to other groups of people.
Our professor told us we can consider different comparisons - for example:

*

*Do Men have asthma at higher rates than Women?

*Do people with University Degrees have asthma at higher rates compared to people without University Degrees?

*Do Men with University Degrees have asthma at higher rates compared to Women without University Degrees?

But as we see, the possible comparisons we can do with only the categorical variables are numerous - and we start factoring in the continuous variables (e.g. Men with University Degrees over the Age of 53 vs. Men with University Degrees over the Age of 53), the number of possible comparisons become even more.
Several of us in the class had this idea of "exploring the data" and seeing what interesting relationships we could find, and then trying to incorporate our findings into new hypothesis questions. But to our surprise, our professor heavily criticized this idea and called it a "classic example of P-Hacking". Our professor told us that for research to be taken seriously - all hypotheses (e.g. which comparisons) must be clearly stated prior to doing the research. Supposedly, this reduces the chances of "getting lucky" and finding some coincidental findings that are the product of chance. Our professor listed several examples of P-Hacking:

*

*Removing and inserting variables into a regression model arbitrarily until desirable p-values are found

*"Fishing" for different hypotheses until one of these hypotheses results in a desirable p-value

While these examples that the professor listed do seem relevant to me - I am not sure if I agree that these practices are "inherently" wrong. For example:

*

*In Machine Learning, there are entire algorithms on "Feature Selection" (e.g. Stepwise Selection, Genetic Algorithm, https://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html) that try different combination of features (and even create new features from old features) until a good model is produced. As I understand, these techniques can be used for a regression model. I understand that these methods can be prone to abuse with regards to P-Hacking (and result in overfitted models that are too complex to understand) - but if large datasets are available, couldn't procedures like Cross Validation be used to rigorously test these models to make sure that a "newly discovered insight" is not occurring by chance, but rather this new insight consistently appears across different levels of variability within the data?


*Regarding the second point, I also agree that "newly discovered trends, patterns and results" can be prone to abuse and can be misleading - but aren't "preconceived hypotheses" equally subject to such abuse and be have the potential to be equally misleading? Suppose I take some two random subset of medical patients (e.g. Men over 34 with University Degrees earning under 44k vs. Men under 34 with University Degrees earning more than 44k ) - I run some statistical comparison tests and find that "Men over 34 with University Degrees earning less than 44k" have asthma at very high rates. In the spirit of Machine Learning, could I not use some Cross Validation themed approach (such as Bootstrap), take repeated random samples of "Men over 34 with University Degrees earning less than 44k" (provided the sample size of this subgroup is large enough) and plot the distribution/histogram of their asthma rate? Could I also not use a similar approach like this to test the statistical significance between groups? Could such an approach not be used to partly determine if P-Hacking is occurring?


*The prof brought up that "if you fish for something long enough, you will eventually find it". By this, the prof meant that testing many different hypotheses will inevitably result in some false hypotheses being accepted and some true hypotheses being rejected. Doesn't the Bonferoni Correction Factor assist in adjusting the statistical significance so that the probability thresholds are more "strict" and result in hypotheses on the "cusp of being accepted" fall on the rejection side?


*And finally, if the entire research procedure, trials and errors, all hypotheses and results are formally stated (e.g. even the results that look "less attractive" with insignificant p-values) - couldn't this also be used to argue the case that P-Hacking and other malicious research practices did not occur?
Can someone please comment this - if my logic is correct, wouldn't this make aspects within Machine Learning and Exploratory Data Analysis fundamentally at odds with P-Hacking? Or can large datasets and Cross Validation be used to partly circumvent and mitigate such issues?

The TLDR
The short answer to your question is no. You can engage in both null hypothesis significance testing (NHST) as well as exploratory data analysis (EDA) without engaging in morally corrupt research practices. The longer answer to that question can be summarized with "it also depends."
A Practical Example
If we remove the frame of formal scientific research, we can apply both of these types of data analysis on everyday life and see why they are both important (and of consequence, why they are not always at odds with each other). Let's say my computer breaks one day. I hypothesize that this is due to the power button being dysfunctional. My null hypothesis would be that pressing the power button should have no effect on the outcome. So let's say I press the button a thousand times and nothing happens. I have now failed to reject the null hypothesis.
But of course somebody who has broken their computer wouldn't just throw their digital baby into the trash bin. They would try to find some way to figure out the problem, and thus would explore other options. That can be done both with formal hypotheses ("perhaps it is because of the charger") or by simply mashing all the buttons until something works ("idk what is going on with my computer but maybe blunt force trauma will do the trick."). You can see where this would be a solution rather than a problem, because now you are combining exploratory and confirmatory methods. Consider the below chart for the scientific method:

You can see that some parts of this method are confirmatory and others are exploratory. Take for example the section that shows where one would "troubleshoot." We do this all the time with broken computers, and that would also apply with our scenario above. You would also engage in updates to your hypotheses ("maybe it is broken because of X, Y, or Z"). Only by exploring further can we figure this out. Drawing up hypotheses from nowhere is a crackpot's idea of science anyway.
Now if we were to inject a p-hacking scenario into this real-life situation, it may look something like the next scenario. Let's say my roommate also has a broken computer and I have no pre-conceived notion of what is causing the issue. So I mash all of the buttons on his keyboard until it decides to turn on. I don't know what actually fixed it, but because one of the buttons was the power button, I tell him after that I already devised that it was the power button the whole time. Obviously this is lying...I didn't fix the button because I knew the cause already, I simply explored around and conceived of the idea after.
This situation is more where you will find issues with blending NHST and EDA. If you decide to dig around until you get something you want, then naturally you cannot say with certainty that you hypothesized it. But that's not to say that you shouldn't engage in EDA to facilitate NHST. It only means that you should engage in other practices, such as reporting your failed hypotheses, including newly developed hypotheses, and providing justification for why EDA was used (usually to motivate a decision on what the causes of a phenomenon are).
Final Remarks
The reason that I suspect he has made this hard distinction in your class is because failing to state some explicit ideas before data analysis can be very problematic if you are going to be a researcher or statistician, and he is probably attempting to get you to think in a more scientifically rigorous way. This isn't inherently wrong, but I do think that stating this in such a rigid way isn't always helpful. Perhaps some others here may disagree with me, but that is my assumption based off my intuition on the subject.
Edit
In the comments, Karolis mentioned the potential issue with using the power button analogy as being a false example because you always get the answer to your outcome. But as I mentioned there, this is actually a probabilistic outcome disguised as a closed solution. I give a "power surge scenario" in the comments, but there are other factors that can influence this outcome (and are not limited to just these instances):

*

*Your battery is dead before you do a certain number of trials. While you are gone, somebody replaces it. You turn it on and suddenly your computer turns on after more trials. You have falsely rejected the null because the button wasn't the issue.

*Your glasses were not on that day and you can't see the buttons clearly. You think you are pressing the power button, but you actually pressed something else and now the computer is on. So your perception was the computer was turned on by the power button, but it was actually something entirely different.

*The computer doesn't turn on because you didn't apply enough pressure when pushing down with your finger. You retain the null hypothesis but this in fact isn't true...you simply didn't push down hard enough on the button.

This is actually in line with what the professor said...getting the laptop to power on or not power on may be purely by chance, but having a solid theory and evidence behind your claim will help reduce the risk of chance outcomes and give you a solid starting point for attacking the problem.
To your question in the comments about cross-validation...you have asked a lot here so I will simply state that cross-validation doesn't become a magic bullet for solving the dilemma. Karolis has provided some reasons for why this can be an issue that I agree with.
