What will be the null hypothesis in this study? The disease X can be treated with either drug A or drug B. Doctors have used one of these two drugs for several years. From their experience, they believe that mortality with both drugs is similar.
A researcher desires to formally compare the two drugs. The researcher plans to retrospectively analyze the mortality in the patients treated with the two drugs. The researcher's study hypothesis states that the mortality with drug A and drug B is not different. The researcher obtains a P-value >0.05 for this comparison and concludes that both the drugs are comparable.
Now, I know that in a clinical trial, such a study hypothesis (stating no difference between two arms) is not acceptable. The establishment of comparability of the two drugs would need an equivalence trial. More commonly, researchers would use a superiority trial. The study hypothesis of the superiority trial would state that one of the drugs is better than the other. The null hypothesis would state that the drugs are not different.
However, in the setting of a retrospective study (when experience makes us believe that the drugs may be comparable),
(1) Can we use a study hypothesis that states that the drugs are similar?
(2) If (1) is acceptable, can the researcher conclude that the drugs are comparable when the P-value is not statistically significant and fails to reject the null hypothesis?
(3) Should we use a study hypothesis which states that one drug is better than the other (similar to a clinical trial) disregarding our experience?

*

*Yes. This is called equivalence testing. The easiest equivalence test to understand is called two one-sided tests (TOST). Briefly, TOST uses one-sided tests to show that neither drug is much better than the other, bounding the difference in some region the investigator deems practically equivalent.



*No. This is a common misinterpretation of what a p-value means. A high p-value can result from a test having inadequate power to detect a difference of interest. In other words, if you look for something and don’t find it, it doesn’t count unless you’ve looked hard enough (where “hard enough” is quantified by a power calculation).



*No. There are methods, like TOST, that allow you to test exactly what you want to test.

