J. André Knottnerus|Peter Tugwell
Multiple testing in big data
era increasing availability accessibility huge complex databases resulting research routinely collected sources 1hofker fu j wijmenga c genome revolution role understanding complex diseasesbiochim biophys acta 2014 may 13 pii s0925443914001306 epub ahead printhttpdxdoiorg101016jbbadis201405002pubmed google scholar 2berger ml doban v big data advanced analytics future comparative effectiveness researchj comp eff res 2014 3 167176httpdxdoiorg102217cer142crossref pubmed scopus 35 google scholar question dealing adjusting multiple statistical testing even challenging key dilemma detect meaningful associations one hand avoiding numerous falsepositive results conceptual technical issues stake conceptual issues several questions posed example multiple testing defined large series tests performed one specific analysis one also take tests account carried previous investigators database latter case registering testing already done past cumulative history integrated new even future analyses extent performing large number tests based less defined hypotheses less problem fishing furthermore scientific justification hypotheses taken account even independently reviewed analysis would implications record linkage context transparently reported avoid publication bias regarding technical side first question still whether use statistical adjustments number tests performed use simple approach unadjusted testing overall additional investigations focused analysis explorative analyses potentially relevant results come forward use technical adjustment way go 3bender r lange adjusting multiple testingwhen howj clin epidemiol 2001 54 343349abstract full text full text pdf pubmed scopus 1886 google scholar related questions problems require innovative thinking methodology development issue glickman et al introduce alternative bonferronitype adjustments optimize error rate control procedures distinguish hypothesis exploratory andor datadriven testing recommend false discovery rate control review explain approach illustrate advantages using two recent health studies related subject assessment validity large databases valkhoff evaluating accuracy disease codes free text identifying upper gastrointestinal bleeding four european electronic health care databases reviewed patient records assessed effect outcome misclassification estimation drugrelated bleeding concluded icd international classification diseasesrelated coding systems good positive predictive value less granular terminology 4lamberts h wood wonca icpc working party icpc international classification primary care oxford university press oxford1987google scholar may require additional strategies also benchimol et al studied validity health administrative databases focusing accurate disease ascertainment using adultonset inflammatory bowel disease ibd identification algorithms data two large cohorts incident patients linked health administrative data found incident ibd could accurately identified within health administrative data compared large database research methodological quality clinical intervention studies especially represented systematic reviews randomized trials much crystallized developed however still many opportunities improvement various phases types research systematic reviews metaanalyses often considered represent highest level evidence hunter et al emphasize contain assessment potential publication bias influenced results authors studied utility funnel plots context based simulation study metaanalysis published literature perioperative mortality abdominal aortic aneurysm repair conclude conventional funnel plots inaccurate way assess publication bias metaanalyses low proportion outcomes propose alternative funnel plot decision making scientifically ethically approving trials equipoise traditionally important concept taken account donovan et al studied doctors considered experienced concept recruiting participants randomized controlled trials rcts results indepth interviews doctors involved six publicly funded trials conclude recruitment trials difficult fragile process suggest training support enable doctors become comfortable important rct concepts including equipoise promote resilient recruitment process nocebo effect namely unpleasant response application sham treatment andor suggestion negative expectations less studied placebo effect koog et al investigated nocebo effect using randomized acupuncture trials include sham notreatment groups systematic review authors suggest nocebo effect clinically meaningful clinicians interested positive effects also cautious negative effects associated acupuncture pragmatic trials attractive looking reallife clinical effects may subject potential biases context dascanio et al emphasize cohort factorial randomized trial design option reduce biases associated unblinded trials tested feasibility approach studying effect treatments low back pain conclude design considered evaluating novel interventions chronic musculoskeletal problems even pragmatic trials observational studies subject bias critical care medicine randomized trials often difficult perform zhang et al evaluated effect sizes experimental treatments based propensity score ps analysis observational studies comparison metaanalysis randomized trials contrary usually seen found observational research likely report less beneficial effects rcts discuss implications finding intervention effects may differ considerably subgroups often challenge clarify differences using data study effects weight loss intervention trial de vos cs applied latent class growth analysis lcga identify distinct subgroups different weight change trajectories evaluating results authors concluded lcga suitable method purpose preparation conduct interpretation research insight actually happens people participate studies important therefore taking recently published articles journal clinical epidemiology 5mccambridge j kypri k elbourne randomization trust overlooked problems experimenting people behavioral intervention trialsj clin epidemiol 2014 67 247253abstract full text full text pdf pubmed scopus 98 google scholar 6mccambridge j witton j elbourne dr systematic review hawthorne effect new concepts needed study research participation effectsj clin epidemiol 2014 67 267277abstract full text full text pdf pubmed scopus 1366 google scholar mccambridge et al highlight view need better guidance empirical investigations research participation effects discuss existing concepts previous studies participating research suggest mixed methods participantcentered research may lead better prevention bias good response rates basis good research participation often difficult achieve especially mental health settings may easy context maclean et al performed large factorial randomized trial examine effect prenotification postcard short teaser message response rates mailed questionnaire bulimia nervosa mental health literacy found prenotification postcardenhanced response rates teaser clinical epidemiological research comorbidity health outcome increasingly central point interest frei et al aimed identify comorbidities greatest impact patientreported health status chronic obstructive disease developed comorbidity index reflect combined impact comcold index based patient data international collaborative chronic obstructive pulmonary disease study conclude new index well suited purpose complements existing comorbidity indices predict mortality important health events also predicted using probability decision trees shown rafiq et al context large general practice gpbased multicenter trial quality improvement chronic kidney disease elderly patients range available candidate predictors decision tree risk score generated validated used routine gp falls assessments often erroneously assumed internal reliability instruments stable across studies systematic review smith cs evaluated homogeneity determinants heterogeneity nine domains european organization research treatment cancer quality life questionnaire eortcqlqc30 domains except cognitive functioning turned internally reliable number predictors heterogeneity identified authors find process electronic submission articles cumbersome time consuming appreciate buckleys strong plea another guideline addition many already launched guideline standard formatting articles submitted peer review reading proposal editors happy say journal clinical epidemiology moving toward procedure specific detailed format requirements needed comply acceptance publication