Data dredging? Testing a new hypothesis based on result from previous hypothesis on the same data I need some advice from statistics experts.
I often encounter in biomedical research field that people do one statistics test upon another on the same dataset.
Let's say we have two groups of patients, some 'sick' and the other 'healthy control', and we are 'exploring' for explanations and therefore have performed statistics tests on many features, corrected for multiple testing and found a few that were significantly different between the groups. Let's say one of these significant features was the presence of a certain bacteria. Then a second hypothesis based upon this is formed which is that patients in the different groups have different levels of antibodies to this bacteria and this is then tested.
I would say that it is not correct to form new hypotheses based on a dataset and then test them in the same dataset. What is your take on this? Is there a way to get away with the above problem?

Testing lots of hypotheses is not a problem in a preliminary study, and forming new hypotheses from those tests is good. However, if the newly formed hypotheses are chosen on the basis that the preliminary data suggest that there is something there, then you really cannot 'test' the new hypothesis using those same data, just as you suspect. Hypotheses selected on the basis of preliminary data should be tested using new data.
The problem that comes from testing a hypothesis with the data that first suggested the hypothesis is that the data are always going to support the hypothesis! After all, that's why the hypothesis was chosen in the first place. The false discovery rate and false positive rate for the procedure will be far higher than the nominal level.
On the specifics that you mention, the situation might be subtly different if the antibody levels were not measured in the original data collection. Measuring the antibody levels afterwards goes some way towards being the new data necessary for testing the hypotheses that were selected from the original study. However, it seems to me that the antibody levels in patients stratified into groups on the basis of the presence and absence of bacteria might tell you about the relationship between infection and antibodies, but it cannot tell you much about the original condition of grouping, sickness.
