The p value is a reﬂection of the probability of a type I error (alpha). This reﬂects the probability that a difference between study groups could have arisen by chance alone. In other words, it is the probability that there is a difference between therapies, interventions, or observed groups when a true difference does not exist.
Historically in the medical literature, a p value of less than or equal to 0.05 was used to determine statistical significance. This reﬂects a probability of 1 in 20 that the null hypothesis was rejected based on the results from the study sample. This p value may be adjusted downward if multiple associations are being tested and the chances of false discovery are high. In genome-wide association studies, in which hundreds of thousands of genetic variants are tested between groups, p values are frequently set at 10−7 (0.0000001).
Beta Error and Power Type II (or beta) error reﬂects the probability of failing to reject the null hypothesis when in reality it is incorrect (i.e., there truly was a treatment effect or a difference between the observed groups). In clinical trials it is important for the investigator to address the beta error, even in the design stage of the study. Study planners should calculate the power (or 1–the beta error) that their study would have to detect an association, given assumptions made about the differences expected between treatments, and design the study size accordingly. Be aware that small clinical trials cited as evidence for “no effect of therapy,” may not have an adequate sample size to address the study question; in essence, the study is not powered to detect the difference.
Confidence Intervals Confidence intervals (CI) provide the investigator an estimated range in which the true statistical measure (e.g., mean, proportion, and relative risk) is expected to occur. A 95% confidence interval implies that if the study were to be repeated within the same sample population numerous times, the confidence interval estimates would contain the true population parameter 95% of the time. In other words, the probability of observing the true value outside of this range is less than 0.05. When evaluating measures of association, such as odds ratio or relative risk with 95% CI, values that include 1 (no difference) are not considered statistically significant.
Meta-analysis One way of improving precision of the effect measure and narrowing the confidence interval is to perform a meta-analysis in which treatment effects from several clinical trials are aggregated to provide a summary measure. Meta-analysis is a favorite tool of the Cochrane database, with which clinicians should be familiar (8). However, there are important considerations in interpreting the meta-analysis, including whether studies were similar enough in their design to be aggregated. Guides for systematic reviews and meta-analysis that involve randomized controlled trials (i.e., the Preferred Reporting Items for Systematic Reviews and Meta-Analyses [PRISMA] statement) and observational studies (i.e., the Meta-analysis Of Observational Studies in Epidemiology [MOOSE] guidelines) are excellent resources for both the investigator and the reviewer (9,10).
Bias Bias is a systematic error in the design, conduct, or analysis of a study that can result in invalid conclusions. It is important for an investigator to anticipate the types of bias that might occur in a study and correct them during the design of the study, because it may be difficult or impossible to correct for them in the analysis.
Information bias occurs when participants are classified incorrectly with respect to exposure or disease. This may occur if records are incomplete or if the criteria for exposure or outcome were poorly defined, leading to misclassification.
Recall bias is a specific type of information bias that may occur if cases are more likely than controls to remember or to reveal past exposures. In addition to establishing well-defined study criteria and accessing complete records, information bias may be reduced by blinding interviewers to a participant’s study group.
Selection bias may occur when choosing cases or controls in a case-control study and when choosing exposed or unexposed subjects in a cohort study. A systematic error in selecting participants may inﬂuence the outcome by distorting the measure of association between the exposure and the outcome. Including an adequately large study sample and obtaining information about nonparticipants may reduce bias or provide information to evaluate potential selection bias.