David N Borg|Keith R. Lohse|Kristin Sainani
Ten Common Statistical Errors from All Phases of Research, and Their Fixes
Griffith University|University of Utah|Stanford University
statistical errors surprisingly common threaten credibility biomedical research13 issue received increasing attention recent years including popular media47 prompting calls statistical reform debates improve statistical practice largely focused choice inferential method particular p values statistical significance testing come scrutiny812 may give applied scientists mistaken impression better statistics simple changing ones inferential methodfor example replacing frequentist ttest bayesian one p value confidence interval transform statistically unsound study statically sound one fact errors threaten studys validity usually occur long researcher calculates p value jeffrey leek roger peng write arguing p value like focusing single misspelling rather faulty logic sentence13 like leek peng believe inappropriate study design unclearly formulated research questions poor data handling lack statistical thinking numerical literacy pose even greater threats science misused p values article draw attention statistical errors occur steps research pipeline present examples 10 common mistakes occur four phases research study design data wrangling cleaning data analysis reporting examples hypothetical based real cases encountered also discuss potential solutions help researchers avoid mistakes study design involves many steps including defining clear research question aligning question suitable study design determining adequate sample size planning variables measured errors steps may result studies flawed virtually useless answering research question interest illustrate several examples problem 1the goal study show equivalence noninferiority study designed superiority trial example thirty athletes randomly assigned two groups ketogenic diet habitual diet 4 weeks researchers aimed demonstrate ketogenic diet cause elevations inflammation compared normal diet authors compared 4week adiponectin marker inflammation levels two groups found significant difference p 50 concluded 4week period consuming ketogenic diet increase inflammation however conclusion follow failure find significant difference proof difference solution authors designed study noninferiority trial aims show one intervention unacceptably worse another14 noninferiority study researchers specify priori noninferiority margina difference would considered small care eg increase adiponectin 05 mgl sample size calculations incorporate noninferiority margin problem 2no sample size planning performed leading results imprecise useful example study aimed assess interrater reliability new test measuring balance stroke patients two raters measured balance 10 stroke patients authors calculated intraclass correlation coefficient icc 95 confidence interval ci 76 23 93 problem estimate icc imprecise interrater reliability could plausibly anywhere insufficient excellent solution authors performed priori sample size calculation reliability studies sample size calculations based achieving sufficient precision icc comparing icc nonzero null value 50 handy online calculator found here15 problem 3the study used measurement scale previously validated example study aimed investigate changes subjective mental fatigue induced netball match play twelve netball players followed competitive season mental fatigue ratings collected match authors asked players report mental fatigue scale made 100mm visual analogue scale 0 none 100 maximal study found significant reductions pre postmatch problem unclear whether scale valid reliable validity means tool measures purports measure reliability means tool gives consistent results repeated measurements thus observed difference may due measurement error rather real effects solution authors used previously validated measure subjective mental fatigue known reliability validity ideally measure demonstrated reliabilityvalidity similar population data collected checked data entry errors consistency eg check men values indicating menstruation pregnancy researchers also typically need create new variablessuch calculating age birthdatesa process introduce novel errors data errors easy overlook undermine studys conclusions following examples show problem 4mistakes data manipulation led 10 reversal coding binary variable example researchers surveyed 700 runners determine sun protection habits initial analysis concluded women older runners runners history skin cancer less likely report consistent use sun protection results opposite would expected scrutiny revealed coding error survey sun protection coded 1 always 5 never scale however data converted binary variable excel values 4 5 erroneously recoded indicating consistent sun protection consistentprotection 1 whereas categories 1 3 recoded indicating lack consistent sun protection consistentprotection 0 fortunately error caught publication solution data cleaning checking manipulation done within statistical analysis program eg sas stata r record changes data kept code variables checked accuracy consistency running statistical models problem 5a data entry error created spurious association example researchers prospectively collected data 150 national collegiate athletic association division college distance runners including sleep habits 1mile running times initial data analysis revealed moderate correlation longer average nightly sleep faster 1mile performances correlation coefficient 55 p 01 however data subsequently plotted data entry error revealed one runner reported best 1mile time 7 minutes 30 seconds also reported relatively short sleep times average 5 hours per night one data point removed correlation sleep 1mile performance time negligible r 15 p 46 given 7 minutes 30 seconds unexpectedly slow population data point rechecked found misentered database fortunately error caught publication solution running formal analyses researchers plot check variables identify outliers data entry errors16 data entry errors fixed outliers remain dataset influence probed sensitivity analyses data analysis involves many steps including understanding visualizing data choosing right statistical approaches data research question testing relevant model assumptions understanding matter following best practices model building correctly interpreting model outputs adequately checking visualizing final models researchers skip steps perform blindly cookbook fashion high chance error several examples illustrate problem 6failure check relevant assumptions statistical model led incorrect inferences example student analyzing data angular rotation head across four different conditions healthy controls individuals mild traumatic brain injury using repeatedmeasures analysis variance anova student found main effect condition statistically significant p 032 manuscript preparation coauthor asked see raw data point student advisor realized variance angular rotation two conditions three times larger conditions angular rotation also positive skew data violated two assumptions anova homogeneity variance specifically sphericity case normality residuals violations sphericity lead inflated type error rates nonparametric test conducted data resulting p value much higher p 24 solution researchers fully understand variables data set running formal statistical models tests descriptive statistics distributional plots histograms generated relevant variables researchers also test assumptions formal statistical tests plan run determine whether alternative statistical approaches required problem 7the wrong statistical approach applied data leading potentially misleading conclusions example study aimed determine influence calcium intake returntoplay softtissue injury sixtytwo amateur australian rules football players followed two competitive seasons data calcium intake injury status recorded researchers fit linear regression model returntoplay outcome variable returntoplay coded 0 uninjured players numbers weeks missed injured athletes n 1262 based model researchers concluded calcium intake related faster healing times average reduction returntoplay 02 weeks per 100 mg calcium intake p 03 problem returntoplay bimodal distributionmany athletes value 0 whereas others value least several weeks linear regression model effectively fits line two clouds points thus observed associations likely reflect effect predictor incidence rather duration injury addition normality homogeneity variance assumptions linear regression model likely violated could lead incorrect standard errors p values solution returntoplay analyzed using injured athletes alternatively researchers could used zeroinflated model capable handling bimodal distribution weeks missed due injury problem 8the data analysis ignored important sources correlation leading p values drastically underestimated hypothetical example researchers conducted randomized clustered trial education intervention aimed preventing injuries five high schools randomly assigned receive education intervention five randomly assigned control condition coaches intervention condition n 10 attended 1hour educational workshop taught specific warmup regimen reduce injuries whereas coaches control condition n 10 received educational email researchers attended teams games matches subsequent year recorded number times warmup routine implemented found intervention coaches implemented routine 54 200 27 games whereas control coaches 30 200 15 games using chisquare test authors concluded difference highly significant p 005 however analysis treats 400 observations independent independent two sources correlation coach measured repeatedly multiple games coaches clustered within high schools taking account sources correlation drastically increase p values since effective sample size much less 400 solution analyzing correlated observations sources correlation must accounted either changing unit observation eg games coaches andor using statistical model handle correlated observations17 researchers completed analyses need accurately summarize analytic approach results translate findings readers unfortunately authors commonly spin results meaning consciously unconsciously mislead readers18 errors reporting particularly insidious often magnified subsequent studies cite spun results problem 9the abstract highlights withingroup results omits betweengroups results obscuring fact groups differ significantly hypothetical example researchers conducted randomized placebocontrolled trial fish oil improving cognitive function stroke patients cognitive function improved average 3 points fish oil group n 20 withingroup change statistically significant 05 level p 043 cognitive function improved 21 points placebo group n 20 withingroup change statistically significant p 087 average betweengroup difference 09 points statistically significant p 47 however authors write abstract fish oil group improved significantly p 05 whereas placebo group p 05 thus fish oil may beneficial improving cognitive function highly misleading presentation results significance within one group imply significant difference groups19 solution reporting results controlled trials abstract present betweengroup comparisons primary conclusions based betweengroup comparisons problem 10closely related analyses single data set reported across multiple papers single report would sufficient hypothetical example researchers interested effects lifelong exercise habits occurrence joint pain later life researchers surveyed 1000 older adults determine current joint pain well exercise habits decade life span three logistic regression models separately run probing effects early life participation swimming running ball sports latelife joint pain researchers found swimming associated reduced risk joint pain running associated increased risk ball sports little effect results published three separate papers without authors informing readers existence analyses problem readers missing important context judge statistics example testing three sports increases overall type error rate splitting analyses also precludes potentially relevant analyses adjusting participation one activity participation another finally harder readers detect discrepancies modelssuch adjustment body mass one sport anotherthat may reflect researcher bias solution researchers always think carefully whether multiple analyses data set warrant multiple papers whether analyses best presented single paper particular statistical approach applied outcome sample single paper may appropriate statistical errors occur phases study study design reporting presented common errors occur along research pipeline errors unrelated misunderstandings misuse p values statistical significance testing although correcting inferencerelated errors important believe change alone minimal impact improve statistical practice applied areas must increase statistical numerical literacy along steps research pipeline