Kristin Sainani|Keith R. Lohse|Paul Remy Jones|Andrew Vickers
Magnitude?based Inference is not Bayesian and is not a valid method of inference
Stanford Health Care|Stanford University|University of Utah|Norwegian School of Sport Sciences|Memorial Sloan Kettering Cancer Center
scandinavian journal medicine science sportsvolume 29 issue 9 p 14281436 letter editoropen access magnitudebased inference bayesian valid method inference kristin l sainani kristin l sainani orcidorg000000030614303x department health research policy division epidemiology stanford university stanford californiasearch papers authorkeith r lohse keith r lohse department health kinesiology recreation university utah salt lake city utah department physical therapy athletic training university utah salt lake city utahsearch papers authorpaul remy jones paul remy jones department sports medicine norwegian school sport sciences oslo norwaysearch papers authorandrew vickers corresponding author andrew vickers vickersamskccorg department epidemiology biostatistics memorial sloan kettering cancer center new york new york correspondence andrew vickers department epidemiology biostatistics memorial sloan kettering cancer center new york ny email vickersamskccorgsearch papers author kristin l sainani kristin l sainani orcidorg000000030614303x department health research policy division epidemiology stanford university stanford californiasearch papers authorkeith r lohse keith r lohse department health kinesiology recreation university utah salt lake city utah department physical therapy athletic training university utah salt lake city utahsearch papers authorpaul remy jones paul remy jones department sports medicine norwegian school sport sciences oslo norwaysearch papers authorandrew vickers corresponding author andrew vickers vickersamskccorg department epidemiology biostatistics memorial sloan kettering cancer center new york new york correspondence andrew vickers department epidemiology biostatistics memorial sloan kettering cancer center new york ny email vickersamskccorgsearch papers author first published 31 may 2019 httpsdoiorg101111sms13491citations 24aboutsectionspdf toolsrequest permissionexport citationadd favoritestrack citation shareshare give accessshare full text accessshare fulltext accessplease review terms conditions use check box share fulltext version articlei read accept wiley online library terms conditions useshareable linkuse link share fulltext version article friends colleagues learn morecopy url share linkshare onfacebooktwitterlinkedinredditwechat people thought earth flat wrong people thought earth spherical wrong think thinking earth spherical wrong thinking earth flat view wronger put together 1 isaac asimov recently diong2 wrote commentary scandinavian journal medicine science sports critiquing study pamboris et al3 correctly interpreting confidence intervals contained zero pamboris et al4 wrote response letter return seeking rebut diongs critique arguing appropriately implementing magnitudebased inference mbi argue based bayesian inference conclusions robust4 neglect however multiple statisticians57 including bayesian statisticians8 9 strongly critiqued even called rejection mbi method inference commentary hope resolve ambiguity concisely explain mbi robust method statistical inference response letter pamboris et al4 correctly note diong2 defined confidence intervals inaccurately however diongs definitional error undermine central argument original authors reached conclusions justified data pamboris et al compared slow dynamic stretching sds fast dynamic stretching fds 18person crossover design3 concluded sds showed greater improvement fds neuromechanical sensorimotor performance differences observed conditions small sample completely consistent random variation due sampling variability show numerous papers sport exercise literature use mbi led authors reach overly optimistic conclusions pamboris et al4 claim conclusions justified interpretation results according recommended mbi procedures mbi established statistical method date mbi never published statistical journal1012 equations formal notation ever provided instead mbi implemented microsoft excel spreadsheets solely available wwwsportsciorg clear documentation version control13 14 pamboris et al4 also fail cite numerous critiques mbi statisticians academic literature59 popular press15 16 arguments neglect mbi questioned outright banned major sports medicine journals17 18 prima facie evidence make one skeptical legitimacy mbi using pamboris et al paper3 motivating example give brief review mbi explain robust method inference 1 mbi method mbi arose understandable desire put emphasis effect sizes estimation mbi classifies results three categories effect size harmful negative trivial beneficial positive pamboris et al3 defined trivial values 02 02 standardized effect size units default values recommended mbis creators10 11 experience values selected vast majority mbi practitioners mbi next assigns probabilities true effect either beneficial positive trivial harmful negative example pamboris et al3 report 74 chance sds improves positional error relative fds table 33 probabilities derived interpretation confidence intervals pvalues demonstrate incorrect mbi excel spreadsheets mbi calculates onesided pvalues standard hypothesis tests5 onesided pvalue benefit gives probability observed data values extreme would arisen intervention beneficial eg true effect 02 example pvalue 010 tells us effect least large one sample 10 chance arising intervention beneficial written recall symbol means given conditioned also note simplicity use data denote observed value values extreme taught introductory statistics classes pvalue tell us probability true effect beneficial beneficial given data yet exactly mbi interprets mbi concludes 10 chance intervention beneficial 90 chance beneficial interpreting pvalues way equivalent interpreting frequentist confidence intervals bayesian credible intervals later show incorrect interestingly pamboris et al appear unaware math underlying spreadsheets state response letter4 inferences nothing whatsoever nhst null hypothesis significance testing fact mbi probabilities calculated based standard hypothesis tests spreadsheets used pamboris et al implement two onesided tests nonzero null values5 shown mathematically welsh knight5 confirm simulated data randomized trial n 10 per group normally distributed outcome sd 10 means 01 0 experimental control groups respectively analyzed data mbi spreadsheet comparing two means xcompare2groupsxls got probability benefit 35603 ran welchs test unequal variances h0 02 onesided benefit sas version 94 got pvalue 064397 1064397 035603 magnitudebased inference provides following qualitative descriptors probabilities 05 unlikely 055 unlikely 525 unlikely 2575 possible 7595 likely 95995 likely 995 likely10 11 mbi also deems effects unclear probabilities harm benefit elevated typically 5 equivalent 90 confidence interval overlapping harmful beneficial ranges accordingly tables pamboris et al3 label effects unlikely possible likely likely unclear 2 statisticians criticized mbi statisticians noted several problems mbi including lacks sound mathematical foundation incorrectly interprets frequentist statistics pvalues confidence intervals gives overly optimistic inferences increasing risk finding spurious effects especially using small samples magnitudebased inference first introduced peerreviewed literature 200610 method later criticized journal statisticians richard barker matthew schofield argued way make probabilistic statements true effectssuch 90 chance intervention beneficialis adopt fully bayesian approach8 also noted mbi draws incorrect overly optimistic inferences confidence intervals shown figure 1 figure 1open figure viewerpowerpoint figure shows appropriate inferences drawn different 95 confidence intervals 90 lower level confidence pamboris paper3 based fall relative harmful trivial beneficial ranges contrasts incorrect inferences mbi draws shaded 90 confidence interval similar confidence interval 169 965 improvement muscle strain sds versus fds paper pamboris et al3 used finding well several possibly beneficial findings justify conclusion sds showed greater improvement fds neuromechanical sensorimotor performance figure based barker schofield8 example pamboris study3 participants experienced average 398 improvement muscle strain sds compared fds condition 90 confidence interval 169 965 figure 1 according mbi methodology gives conclusion sds likely beneficial compared fds correct interpretation confidence interval assuming 169 within trivial range sds harmful relative fds figure 1 concluding sds showed greater improvement fds neuromechanical sensorimotor performance warranted 2015 another pair statisticiansalan welsh emma knightagain raised alarm mbi5 mathematical formulas implemented mbi spreadsheets published anywhere welsh knight painstakingly extracted algorithms cells excel spreadsheets showed among issues method leads inflated rates type error recall type errors falsepositive errors response critique mbis creators published 2016 paper journal sports medicine claiming mbi better type type ii error rates standard hypothesis testing19 claim dubious face value since type type ii error necessarily trade indeed statistician kristin sainani coauthor paper used math simulation definitively disprove claim6 showed typical usage mbi yields unacceptably high type error rates mbis sample size calculators tuned maximize type error crux welsh knights sainanis arguments mbi inflates risk concluding benefit ineffective interventions like statistical significance testing mbi provides thresholds evidence instead highlighting p 005 p 001 mbi practitioners highlight possible likely effects example pamboris et al3 call one likely several possible differences sds fds see table 33 use findings justify conclusion sds showed greater improvement fds neuromechanical sensorimotor performance mbis evidentiary thresholds extremely weak20 using math simulation estimate thatfor way mbi typically applied literaturethe possible bar comparable using significance threshold high p 060 likely bar comparable using significance threshold high p 0251 means intervention completely ineffective mbi practitioners find falsepositive possible effects much 60 time falsepositive likely effects much 25 time unfortunately mbi practitioners informed levels evidence weak pamboris et al instance insist conclusions robust4 fact evidence pamboris et al3 present support claims weak though pamboris et al3 present standard pvalues comparisons easy calculate confidence intervals provided pvalue gives important information level evidence data provide regardless whether one performs binary significance test example pamboris et al3 found sds likely beneficial effect muscle strain compared fds backcalculated twosided pvalue comparison null hypothesis true effect 0 pt17398324 t17398324 0242 means truly difference conditions would seen difference 398 bigger nearly quarter studies chance note betweencondition comparisons eg torque associated even higher pvalues meaning represent even weaker levels evidence claims mbi studies propagated literature definitive findings discussion section pamboris et al3 cite previous study21 say found decrease muscle fascicle strain combined increase muscle stiffness reference mbi study provided similarly weak levels evidence conclusion yet findings qualified based weak evidence instead presented conclusive claims wrong falsepositive errors mbis creators argued various critiques peerreviewed literature personal website primarily trafficked sports scientists statisticians22 creates illusion active debate fact mbi support mainstream statistical community important note additionally counterarguments never questioned math presented statisticians5 6 8 9 rather response various critiques mbis creators focused attacking classical frequentism shifted description mbi response welsh knights paper5 wrote mbi quite possibly ideal frequentistbayesian hybrid12 recently begun describe mbi purely bayesian method avoid critiques mbi website write advise researchers describe inferences magnitudes legitimate reference bayes dispersed uniform prior submitting manuscripts journal banning mbi22 pamboris colleagues response letter adopts tactic4 claiming inferences bayesian despite never describing methods bayesian original paper3 sections follow explain bayesian analysis mbi valid bayesian analysis 3 bayesian analysis previously described pvalues give probability observed data extreme values given null hypothesis true effect beneficial true effect 0 pvalue probability true effect beneficial given data latter quantity would often like know cannot infer probability directly data rather apply bayes rule requires additional information red additional information numerator called prior probability reflects beliefs true effect prior conducting study probabilities data called likelihood function combining likelihood function prior probability gives posterior probability want know simplicity often written bayesian analysis data calculate single probability alone instead calculates full probability distribution using posterior probability distribution bayesians compute credible intervals give probability true effect falls within certain range example 90 bayesian credible interval 15 50 tells us 90 chance true effect lies 15 50 interpreting pvalue bayesian posterior probability equivalent interpreting confidence interval bayesian credible interval finally important note posterior distribution calculated make necessarily valid reliable prior distribution represents assumption nature reality model assumption prior distribution must explained justified tested 4 mbi bayesian analysis mbi spreadsheets make inferences solely based frequentist statistics5 implement type bayesian analysis mbi interpret frequentist statistics bayesian statistics section explain acceptable approach defining feature bayesian analysis incorporation prior probability distribution previously mentioned frequentist statistics give probabilities data given hypothesis eg probability data intervention equivalent control 10 problem would prefer obtain probability hypothesis given data eg 90 chance intervention beneficial bayesian statistical methods calculate probabilities incorporating additional information beyond data alone provide additional information called prior probability distribution described choice prior greatly affect results particularly dealing small samples pamboris colleagues equate mbi bayesian inference default prior4 default prior sufficient detail scientific paper example common default choice comparing means cauchy probability distribution looks like normal distribution area tails even single cauchy distributionthere infinite number determined distributions scale parameter bayesian employing cauchy prior would perform sensitivity analysis determine choice scale parameter affects results mbi practitioners specify prior reflect choice prior might affect results actually incorporate prior magnitudebased inference attempts simply bypass prior rather using central bayesian formula posterior likelihood prior mbi instead assumes posterior likelihood cases thus frequentist pvalues confidence intervals given direct bayesian interpretation mbi pvalues interpreted bayesian posterior probabilities equivalently confidence intervals interpreted bayesian credible intervals several problems 1 mbi users never required specify justify prior think implications perform sensitivity analyses gauge effect choice results 2 simple problems comparing means known standard deviation confidence intervals equivalent credible intervals pvalues posterior probabilities one assumes flat uniform prior8 23 flat priors unrealistic applications biomedicine sports science explain magnitudebased inference described bayesian analysis flat prior10 writings mbi fail inform users implications flat prior flat prior means literally effect sizes equally likely unrealistic assumption biomedicine implies almost every intervention substantial effect many impossibly large effects effects equally likely2325 flat prior almost always lead overly optimistic inferences becauseby starting assumption intervention almost certainly works nontrivial effectit stacks deck favor finding nontrivial effect zwet23 writes uniform prior realistic context biomedical research consequently use leads overestimation magnitude regression coefficient overconfidence sign greenland24 states flat prior generally nonsensical scientific terms suboptimal frequentist bayesian decision inference flat priors serve bound results optimized sensible priors instance pamboris et al3 report stretching increased isometric torque measure muscle force 91 95 newton meters nm flat prior implies likely stretching would reduce torque zero conversely create superhuman strength poststretching torque 1000 nm obviously completely unrealistic assumption flat priors particularly unrealistic sports science empirical data show effect sizes sport exercise science predominantly small2628 fact mbi specifically advertised tool needed small effects predominate sports science22 greenland demonstrated frequentist pvalues confidence intervals viewed providing bounds bayesian posterior probabilities assuming prior symmetric around null25 indeed believe mbi probabilities provide upper limit bayesian posterior probability words mbi returns 50 chance beneficial actually means 50 maximum chance intervention beneficial choice prior assuming priors symmetric around null lead lower probability means using mbi currently implemented almost always present overly optimistic picture interventions effectiveness easy see interpreting frequentist confidence intervals bayesian credible intervals equivalently interpreting pvalues bayesian posterior probabilities lead overly optimistic inferences imagine 95 confidence intervals medical literature suddenly interpreted bayesian credible intervals mbi would allow do2931 would imply least 975 significant beneficial results would represent true positives runs directly counter know true based current reproducibility crisis biomedicine3235 finally important recognize mbis evidentiary thresholds eg likely possible represent weak levels evidence whether taking frequentist bayesian perspective likely beneficial effects typically correspond bayesians would consider anecdotal evidence effect36 5 practical issues use mbi addition theoretical issues mbi believe use encourages poor statistical practices researchers performing mbi analyses overwhelmingly use mbi excel spreadsheets unfortunately black box approach spreadsheets promote means researchers neither exposed statistical theory claimed underpin mbi bayesian inference computational procedures involved generating test statistics creating confusion regarding methods used results produced example response letter pamboris et al4 make bizarre claim used prepost parallel groups trial spreadsheet calculation tool compares difference pre postchanges slow fast dynamic stretching condition thus eliminating type error4 emphasis added statement erroneoustype error eliminated comparing pre postchanges also reference mbi bayesian analysis default prior without explanation implications suggests lack understanding bayesian statistics finally claim inferences nothing whatsoever nhst4 suggests lack awareness excel spreadsheets implement standard hypothesis tests mbi encourages lack critical thinking fostering instead application statistical rituals37 rather exercising judgment validity statistical methods researchers instead heedlessly transcribe results generated calculations mbi spreadsheets mbi alone operating black box indeed many programs allow users implement statistical methods users understand mbi critically lacks rigorous review documentation mentioned mbis authors never presented mathematical formulation method method never published statistical journal mbis supporting documentation also limited tracking workflow various dependencies within spreadsheets confusing timeconsuming subject following oftenverbose instructions explanationsa note attached one spreadsheet cell 800 words long contrasted statistical software code documentation reviewed critiqued published mainstream literature eg journal statistical software example cookbook approach transformation variables reviewing numerous mbi papers noticed authors commonly logtransform data little explanation given deal nonuniformity errors3843 phrase conventional statistical language google search reveals used wwwsportsciorg almost exclusively mbi papers sports medicine literature cases indication authors actually examined data distributions checked assumptions normality homogeneity variances warrant log transformation sometimes appropriate logtransform done indiscriminately without justification log transformation make results difficult interpret eg would mean say certain type warmup improves log race time 08 use logs also make harder check simple numbers thus making easier numerical errors inconsistencies go undetected pamboris et al3 logtransformed two variables explanation choices log transformation makes tables extremely difficult follow believe introduced inconsistencies inferences example muscle strain changed 3800 sds condition see table 1 1320 fds condition see table 2 whichpresumably due log transformationssomehow translates betweencondition difference 3980 see table 33 3980 reported alongside standardized effect size 050 explanation standardized effect size calculated worryingly confidence interval raw data 3980 5670 yields different inferences confidence interval effect size 050 089 backcalculated pvalue former 024 latter 032 demonstrating inconsistency furthermore use effect size confidence interval directly calculate mbi beneficial probability follows pt17050020089175 72 match value given table 80 finally unclear confidence intervals symmetric confidence intervals built