Is this p-hacking? I'm currently looking into the gender pay gap using data from glass door (found via kaggle). The dataset has columns for gender, age, performance evaluations of employees, seniority, pay etc.
For context: I have learned a lot of Data Science/Machine learning/programming over the past few years, and am just doing a few of my own basic portfolio projects for practice, before applying for jobs.
I have done a fairly naive t-test, comparing average pay for men vs average pay for women. I am now looking to add in controls, comparing similar age groups, seniority, education level etc. I want to do more t-tests, as well as looking at a chi-square distribution and/or ANOVA.
As I do multiple tests A/B tests, I want to avoid p-hacking. I have a few hypotheses, for example I expect the pay gap to be greater for older age groups. But this is mostly exploring the data, I don't have a single hypothesis I am looking to prove for the entire study, nor do I have a political agenda.
I'm not sure it would really count as p-hacking as long as I choose which comparisons to make, and report everything. I would think it's only p-hacking if I selected which t-test results to report to help prove a hypothesis. Is this fair?
And another question (forget my data for a moment), with ANOVA, as it compares multiple groups at once to look for significance, is this not p-hacking?

P-hacking implies an agenda to prove a particular type of hypothesis, and in furtherance of that agenda, several related hypotheses are tested. Then if one of them is found to be "statistically significant", only that test is reported with only its p-value. For instance, if I'm claiming to be psychic, I might test the hypothesis that I can guess the color of a card better than chance, and the hypothesis that I can guess the card value, etc. Then if any of those tests is statistically significant, I ignore the others. This ignores the fact that the total probability of a false positive over any of the hypotheses is larger than the probability of a false positive for any particular one. I.e P(A or B) > P(A).
Here, you don't seem to have an agenda, and you're probably not publishing your results, so you shouldn't worry too much about p-hacking, other than the fact that the more hypotheses you check, the higher the probability of getting at least one false positive. If that concerns you, what you can do is choose an overall alpha value (how much of a probability you're willing to accept for any of the tests yielding a false positive), and then choosing smaller adjusted alphas for the individual tests to get the overall false positive probability sufficiently low. One method that is a bit overly conservative (that is, it's not quite correct, but it's erring on the side of avoiding false positives), but simple to implement, is to divide overall alpha by the number of tests (or multiply the p-values by the number). For instance, if you end up testing $100$ combinations, and one of them has a p-value of $0.001$, you can treat that as if its p-value is $0.1$ (if you're wondering "Wait, does that mean I can get a p-value greater than $1$?", well, that's part of what I was talking about of this being an approximation).
While ANOVA tests involve multiple comparisons, they ultimate output only one p-value. The term "p-hacking" is similar to "cherry-picking". You can't cherry-pick if you only have one cherry.
