Is "Exploratory Data Analysis" Fundamentally At Odds With "P-Hacking"? In our Statistics Class, we are learning about something called "P-Hacking" (also called "Data Snooping", "Data Dredging", etc.).
Here is the example we are working with in our class - we have a large dataset (e.g. over 5 million rows of data) that contain medical data on patients (each row is a medical patient). This includes information such as their age, gender, height, geographical location of residence, weight, income, highest obtained education level - and whether or not they have asthma. We are discussing strategies to perform analysis on this dataset - for example, do certain groups of people have asthma at higher rates compared to other groups of people.
Our professor told us we can consider different comparisons - for example:

*

*Do Men have asthma at higher rates than Women?

*Do people with University Degrees have asthma at higher rates compared to people without University Degrees?

*Do Men with University Degrees have asthma at higher rates compared to Women without University Degrees?

But as we see, the possible comparisons we can do with only the categorical variables are numerous - and we start factoring in the continuous variables (e.g. Men with University Degrees over the Age of 53 vs. Men with University Degrees over the Age of 53), the number of possible comparisons become even more.
Several of us in the class had this idea of "exploring the data" and seeing what interesting relationships we could find, and then trying to incorporate our findings into new hypothesis questions. But to our surprise, our professor heavily criticized this idea and called it a "classic example of P-Hacking". Our professor told us that for research to be taken seriously - all hypotheses (e.g. which comparisons) must be clearly stated prior to doing the research. Supposedly, this reduces the chances of "getting lucky" and finding some coincidental findings that are the product of chance. Our professor listed several examples of P-Hacking:

*

*Removing and inserting variables into a regression model arbitrarily until desirable p-values are found

*"Fishing" for different hypotheses until one of these hypotheses results in a desirable p-value

While these examples that the professor listed do seem relevant to me - I am not sure if I agree that these practices are "inherently" wrong. For example:

*

*In Machine Learning, there are entire algorithms on "Feature Selection" (e.g. Stepwise Selection, Genetic Algorithm, https://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html) that try different combination of features (and even create new features from old features) until a good model is produced. As I understand, these techniques can be used for a regression model. I understand that these methods can be prone to abuse with regards to P-Hacking (and result in overfitted models that are too complex to understand) - but if large datasets are available, couldn't procedures like Cross Validation be used to rigorously test these models to make sure that a "newly discovered insight" is not occurring by chance, but rather this new insight consistently appears across different levels of variability within the data?


*Regarding the second point, I also agree that "newly discovered trends, patterns and results" can be prone to abuse and can be misleading - but aren't "preconceived hypotheses" equally subject to such abuse and be have the potential to be equally misleading? Suppose I take some two random subset of medical patients (e.g. Men over 34 with University Degrees earning under 44k vs. Men under 34 with University Degrees earning more than 44k ) - I run some statistical comparison tests and find that "Men over 34 with University Degrees earning less than 44k" have asthma at very high rates. In the spirit of Machine Learning, could I not use some Cross Validation themed approach (such as Bootstrap), take repeated random samples of "Men over 34 with University Degrees earning less than 44k" (provided the sample size of this subgroup is large enough) and plot the distribution/histogram of their asthma rate? Could I also not use a similar approach like this to test the statistical significance between groups? Could such an approach not be used to partly determine if P-Hacking is occurring?


*The prof brought up that "if you fish for something long enough, you will eventually find it". By this, the prof meant that testing many different hypotheses will inevitably result in some false hypotheses being accepted and some true hypotheses being rejected. Doesn't the Bonferoni Correction Factor assist in adjusting the statistical significance so that the probability thresholds are more "strict" and result in hypotheses on the "cusp of being accepted" fall on the rejection side?


*And finally, if the entire research procedure, trials and errors, all hypotheses and results are formally stated (e.g. even the results that look "less attractive" with insignificant p-values) - couldn't this also be used to argue the case that P-Hacking and other malicious research practices did not occur?
Can someone please comment this - if my logic is correct, wouldn't this make aspects within Machine Learning and Exploratory Data Analysis fundamentally at odds with P-Hacking? Or can large datasets and Cross Validation be used to partly circumvent and mitigate such issues?

Here is my take at the answer.
The problem with exploratory analysis
Your professor is correct - exploratory analysis, if done for long enough, will always find some trend that will seem interesting. You can easily check this yourself - just fill the data you had in class with random numbers (or shuffle it between participants so that all the real trends are gone) and perform an exploratory analysis. If you will use hypothesis testing with a traditional 0.05 p-value cut-off you should find that around every 20th result appears to be significant. But this cannot be true, since the data now is a random sea of numbers. Hence all these "findings" are false positives.
The logic is quite simple - you cannot use such a procedure (extensive exploratory analysis) to claim you have found something when the same procedure would also find something where there is nothing to find.
The best way to address the issue
Of course in science everyone is doing exploratory analyses all the time and they are wonderful. Calculating one predefined hypothesis only would be a waste of scientific resources. Imagine we collect 1000 biopsies from cancer patients and perform gene expression analysis (which can be an experiment worth hundreds of thousands of dollars). And then we only test if one pre-selected gene is different between cases and controls. If not we throw the dataset away (because not doing so would be performing an exploratory analysis) and collect another sample to test another hypothesis. Nobody does that.
The way to deal with the issue is to have a "confirmatory" dataset which is then used in order to check if the results found via exploratory phase can be replicated. In an ideal case this confirmatory dataset should come from a separate collection of samples - not merely on collection split into two (a.k.a. "tain" and "test" in machine learning). However, splitting into two is still better than having no dataset for confirmation.
So the whole procedure looks like this:

*

*Collect a sample of data for exploratory analysis.

*Explore the trends in the data by trying anything you like to try.

*Out of the things you found - select a few that are strongest and most interesting.

*Collect another sample for confirmatory analysis.

*Perform a predefined test on confirmatory dataset based on what you found in step 2.

Another way of dealing with the issue
Another way to deal with the issue, as you have mentioned yourself, is to do a correction for multiple testing. So if in the exploratory phase you performed 100 hypothesis tests you then use something like Bonferroni correction in order to adjust the p-values such that, instead of every 20th being significant, the chance of at least one significant result out of this 100 (in a data with no real trends) would be 5% (one in 20).
The issue here of course is that it is hard to track the number of performed tests in exploratory phase. Also exploration involves a lot of peaking at the data. Say you do some kind of scatter plot and you see that there is a difference between adults and children. Did you perform one test here? No. Surprisingly, you might have performed 10 tests just by one glance. Probably, unknowingly, you just checked for differences between men and women, for differences between different socioeconomic status, etc etc. And you just formally checked one that had the most chance of turning out significant. So in the end it's hard to estimate how to correct for multiple testing.
On top of that many real world experiments nowadays (i.e. that gene expression example) involve multiple testing even in exploratory phases. For example imagine you are looking which genes express differently between cases and controls, between cases and controls stratified by age, between cases and controls stratified by sex, etc. Here in each stage you are checking thousands of genes at once, and you should do some form of multiple testing correction (probably something like FDR), in each exploratory step.
So multiple testing is only valid in quite restricted exploratory analyses when you know how many tests you perform. Otherwise it goes out of hand.
Is cross validation an answer
Just like in machine learning you can't train a model on full data and then check it by doing cross validation here you cannot do exploratory analysis on whole dataset and then cross-validate the hypothesis that were significant. If the whole dataset shows a trend (real or perceived) this already predisposes all the subset of that dataset to follow this "detected" pattern.
So this tells us that we have to split the data into parts before doing exploratory analysis and then perform the exploration only on one part and test on the others. But is there a need for multiple parts as in cross validation, or is one "confirmatory" dataset enough - that is a good question. I think the answer is - the more confirmations, the better. It should boil down to the probability of accepting a false-positive. If something replicates on one confirmatory dataset with a p-value of 0.05 - there is a 5% chance of a false positive. If it replicates on two the chance is 0.25%. But probably we can achieve the same thing by simply adjusting our p-value for confirmation to be below 0.05. Also here we should remember that a real confirmation comes from a different sample altogether, not the same sample split into multiple parts.
Why is this not a problem with predefined hypotheses
Well, no. There is a difference between testing a single hypothesis and fitting a single classifier. Classifier will go through an optimisation stage and will try to fit the data as best as it can. A predefined hypothesis in this case would be more similar to the final model of a classifier. If you have a final model (say a neural network with all weights set in place) - you can test it's performance on a dataset once, without cross validation, as there is nothing to cross-validate. Same idea here. When you have a pre-defined hypothesis it is not adapting to the data anymore. It's just testing how likely that effect is to be obtained by chance.
Hence, predefined hypothesis are not potentially misleading, when calculated correctly. Also note that sample size does not change the probability of a false positive - you will have same chance to get a significant result on random data with 10 samples and with 1,000,000 samples. The main thing sample size gives you is statistical power - that is it increases the chance of detecting TRUE trends.
Why is stating all the procedures and all the results not an answer
Think about the proposed exploratory experiment on random data. We can do 1000 hypotheses and you would find around 50 significant trends. We can then publish these results of our exploration and say that we did a 1000, describe each of them, and we found 50 to be significant and showcase these. But all these are false positives. So merely describing how we p-hacked does not prevent us from p-hacking and there is no way to judge which of these results are true, if any.
