Harlan M. Krumholz
Mathematical Models and the Assessment of Performance in Cardiology
Yale New Haven Hospital|Yale University|Qualidigm (United States)
homecirculationvol 99 16mathematical models assessment performance cardiology free accesseditorialpdfepubaboutview pdfview epubsections toolsadd favoritesdownload citationstrack citationspermissions shareshare onfacebooktwitterlinked inmendeleyreddit jump tofree accesseditorialpdfepubmathematical models assessment performance cardiology harlan krumholz harlan krumholzharlan krumholz section cardiovascular medicine department medicine section chronic disease epidemiology department epidemiology public health yale university school medicine new conn yalenew hospital center outcomes research evaluation new conn qualidigm middletown conn originally published27 apr 1999httpsdoiorg10116101cir99162067circulation 19999920672069we entered new era medicine physicians longer granted assumption excellence studies demonstrated marked variability practice medicine1 revealed abundant opportunities improvement care patients studies accentuated pressure exerted payers public profession assume greater responsibility delivering highquality carein era accountability ratings rankings proliferate public payers seek information performance physicians hospitals health plans consequences rankings profound marketing departments popular press seize results hospitals ranked highly tend tout status advertisements whereas less favorable ratings hope avoid identified hazardsthe interest rankings particularly strong cardiovascular medicine cardiovascular diagnoses represent substantial proportion highvolume highcost admissions hospitals statewide national efforts focused institution physicianspecific outcomes cardiovascular procedures acute myocardial infarction23 several areas bypass surgery mortality rates common basis qualityofcare rankings new york bypass surgery ratings individual surgeons based mortality rates4 highly publicized press studies suggest public ratings stimulated improvements care outcomes patients although evidence indirect5because patients randomly allocated different physicians hospitals comparison outcomes among various sites challenging differences referral patterns catchment areas may result different populations institution consequently comparison crude outcome rates may misleading mortality rate may interpreted differently depending profile patients operative mortality rate 2 bypass surgery may better expected hospital operates older unstable patients worse expected hospital accepts younger elective patients accounting differences patient characteristics key appropriate comparisonin background attempts compare performance among institutions individuals stand mathematical regression models intended level playing field particular logistic regression model emerged accepted method analysis6 type model appropriate evaluation binary outcome mortality provide estimated risk patient easily calculated contemporary software logistic regression model developed possible characterize clinical status group patients calculating mean individual predicted rates particular outcome calculation gives overall prediction expected outcome rate based patients characteristics thus given group patients possible compare actual outcome rates predicted outcome ratesthe validity approach depends mathematical model however adequacy model always readily apparent data entered results produced computer outputs may reveal problems models unless specific diagnostic calculations performed unfortunately physicians lack training critique model physicians seem accept approach survey cardiologists new york revealed 67 found comparison cardiac surgeons adjusted mortality rates accurate somewhat accurate7 conversely substantial minority cardiologists expressed doubts methodology third reporting findings accurate allgiven importance accountability frequent use models rate physicians hospitals need focus attention evaluate impact various approaches many issues remain unresolved minimum standards make model acceptable compare outcomes often model updated revised confident models separate real differences random variation sample sizes required efficient models least susceptible intentional manipulation gaming models important remain black boxes profession need increase literacy regarding riskadjustment approaches pursue methodologically rigorous research address gaps knowledge themin issue circulation ivanov colleagues address important issue model selection revision focusing coronary artery bypass graft surgery8 issue relevant institutions organizations using model developed several years ago produce estimate riskadjusted outcome model need updated gained changing modelto address issue authors consider 3 options first approach use model developed similar population without change readymade model second approach use variables previous model recalculate equation data generate new coefficients new coefficients presumably appropriate population opposed population model originally developed improve predictive ability model without need change approach data collection finally third approach develop new model possibility using new variablesthe authors use standard approaches determine best logistic regression model first compare calibration models measure well predicted model fits actual data good model predict mortality levels severity regression models may accurately predict risk death patients either end spectrum disease severity models tendency predict higher risk death observed patients least severe illness lower risk death observed ill patients model fit actual data levels severity unsatisfactory needs modificationthe use model fit actual data favor hospitals physicians admit certain types patients consider 2 hospitals identical performance hospitals virtue sizes referral areas admit different types patients one hospital admits mostly patients ill whereas hospital admits patients catastrophic illness poorly calibrated model create perception different performance 2 hospitals hospital cares mostly verylowrisk patients may predicted mortality rate higher observed mortality rate model tends overestimate risk low end severity meanwhile hospital cares ill patients may predicted mortality rate underestimates risk mortality leading higher observed predicted mortality rate hospitalbecause calibration model marked effect performance measurement evaluated reported model graphical displays model fit useful quantitative goodnessoffit tests intended determine whether observed data deviate significantly fitted model unfortunately goodnessoffit tests often relatively low power detect model problems9 test demonstrates poor fit model clearly calibrated however test able detect marked conflicts fitted model data problems gross may overlookedin ivanov study investigators found existing model derived sample patients 1991 1993 good calibration significantly poor fit predicted observed results newer models acceptable calibration results illuminate one issues evaluation models case poor calibration may indicate inferior model rather indicating model fails ends spectrum patients clinical severity poor calibration case may principally result improvement time surgical outcomes hospitals may decreased mortality rates good model 5 years ago would tend predict higher mortality rates currently observed models based older data would overestimate contemporary mortality risk improved performance outcomes would degrade overall calibration model using current data would necessarily affect utility assessing relative performance hospitals lack calibration occurs equally across spectrum patient severity illnesssimilarly national model uses data single hospital group hospitals may well calibrated overall performance different sample model derived lack calibration may indicate single hospital performs better expected information important would unfortunate lose perspective recalibrating modelall models calibrated developed however value calibration applied samples depends purpose model model intended give good estimate outcome patient counseling calibration critical model intended provide information relative performance institutions provide comparison time lack calibration readymade model another data set may indicate important difference performancethe investigators also evaluated discriminative ability various models discrimination measures models ability separate patients different outcomes ivanov study investigators determined well model differentiated survived died model perfect discriminant ability patients died would higher predicted mortality patients survived found small statistically significant improvements discriminant ability recalibrated new models compared readymade model impact small differences knownfortunately authors also looked practical impact differences calibration discrimination assessing change surgeon rank newer models found surprisingly little change effort among 14 surgeons evaluated change rank occurred among top 7 3 models recalibration surgeons ranked 8 9 readymade model switched places new model surgeon ranked 13 readymade index moved 2 places numbers 10 11 moving placedespite relatively unimpressive change rankings authors conclude existing model episodically recalibrated compared newer models recommendation vague perhaps indicating uncertainty best course action seems prudent examine models periodically several questions remain old model revised replaced incremental benefits costs models principal purpose affect assessment sole purpose model counsel patient calibration discrimination paramount model updated frequently provide patient current estimate given evolution medical surgical techniques model meant comparison updating model lose historical perspectivealthough article makes valuable contribution directing attention value recalibrating remodeling also many important issues deserve attention regard riskadjustment models even best models explain relatively small proportion variability outcomes remains unknown whether riskadjusted mortality rates indicator quality different models result profoundly different impressions quality among institutions1011 models best identifying quality care even model appropriate sample sizes would required distinguish signal noise another important issue regards reliability data collection possibility gaming model increasing severity complete documentation cost data collection also becoming increasingly important leading greater interest parsimonious models ones derived solely existing data information systemsthe shift toward accountability help patients methods assessment must scientifically sound choices defensible risk models continue utility temptation use available unproven approaches assess quality must resisted study ivanov et al directed attention toward practical issue model selection confronts many organizations research needed determine validity riskadjusted outcomes measure quality best approaches account nonrandom allocation patients across sites given state art seems prudent use comparative outcome information riskadjustment models primarily stimulate benchmarking efforts generate hypotheses improve carereprint requests harlan krumholz md yale university school medicine 333 cedar st room ie61 shm new ct 065208025 opinions expressed editorial necessarily editors american heart association references 1 wennberg je gittelsohn j small area variations health care delivery science1973 18211021108crossrefmedlinegoogle scholar2 romano ps luft hs rainwater ja zach ap report heart attack 19911993 sacramento calif california office statewide health planning development 1997google scholar3 pennsylvania health care cost containment council pennsylvanias guide coronary artery bypass graft surgery 19941995 harrisburg pa 1998google scholar4 hannan el kilburn h jr odonnell jf lukacik g shields ep adult open heart surgery new york state analysis risk factors hospital mortality rates jama1990 26427682774crossrefmedlinegoogle scholar5 peterson ed delong er jollis jg muhlbaier lh mark db effects new yorks bypass surgery provider profiling access care patient outcomes elderly j coll cardiol1998 32993999crossrefmedlinegoogle scholar6 hosmer dw lemeshow applied logistic regression new york ny john wiley sons 1989google scholar7 hannan el stone cc biddle tl debuono ba public release cardiac surgery outcomes data new york new york state cardiologists think heart j1997 13411201128crossrefmedlinegoogle scholar8 ivanov j tu jv naylor cd readymade recalibrated remodeled issues use risk indexes assessing mortality coronary artery bypass graft surgery circulation1999 9920982104crossrefmedlinegoogle scholar9 hosmer dw hosmer le cessie lemeshow comparison goodnessoffit tests logistic regression model stat med1997 16965980crossrefmedlinegoogle scholar10 iezzoni li ash shwartz daley j hughes js mackiernan yd judging hospitals severityadjusted mortality rates influence severityadjustment method j public health1996 8613791387crossrefmedlinegoogle scholar11 iezzoni li ash shwartz landon mackiernan yd predicting inhospital deaths coronary artery bypass graft surgery different severity measures give different predictions med care1998 362839crossrefmedlinegoogle scholar previous back top next figuresreferencesrelateddetailscited claeson sterner attergrim j khajanchi kumar v saha gerdin wrnberg 2018 assessment predictive value international classification diseases injury severity score trauma mortality urban india journal surgical research 101016jjss201803071 229 357364 online publication date 1sep2018 gupta yeh r tamisholland j patel guyton r klein l rab kirtane 2016 implications public reporting riskadjusted mortality following percutaneous coronary intervention jacc cardiovascular interventions 101016jjcin201608012 920 20772085 online publication date 1oct2016 kong c guest g stupart faragher chan watters 2013 recalibration validation preoperative risk prediction model mortality major colorectal surgery diseases colon rectum 101097dcr0b013e31828343f2 567 844849 online publication date 1jul2013 manktelow b seaton field draper e 2013 populationbased estimates inunit survival preterm infants pediatrics 101542peds20122189 1312 e425e432 online publication date 1feb2013 chevreul k prigent durandzaleski steg p 2012 lay media ranking hospitals reflect lower mortality treating acute myocardial infarction archives cardiovascular diseases 101016jacvd201205007 10510 489498 online publication date 1oct2012 madan p elayda lee v wilson j 2011 riskprediction models mortality coronary artery bypass surgery application individual patients international journal cardiology 101016jijcard201002005 1492 227231 online publication date 1jun2011 glance l osler mukamel meredith w wagner j dick 2009 tmpmicd9 annals surgery 101097sla0b013e3181a38f28 2496 10321039 online publication date 1jun2009 reid c billah b dinh smith j skillington p yii seevanayagam mohajeri shardey g 2009 australian risk prediction model 30day mortality isolated coronary artery bypass ausscore journal thoracic cardiovascular surgery 101016jjtcvs200903020 1384 904910 online publication date 1oct2009 staudinger staudinger b ostermann h grabenwger tilg b 2008 model based decision making cardiac surgery encyclopedia information technology curriculum integration 1040189781599048819ch094 587598 barbini e cevenini g scolletta biagioli b giomarelli p barbini p 2007 comparative analysis predictive models morbidity intensive care unit cardiac surgery part model planning bmc medical informatics decision making 10118614726947735 71 online publication date 1dec2007 staudinger staudinger b ostermann h staudinger r tilg b 2008 quality assurance cardiac surgery setting austrian heart register standard hospital information systems suitable quality assurance journal 101002qaj415 112 151159 online publication date 1jun2007 brunelli varela g berrisford r rocco g 2007 audit quality control performance thoracic surgerya european perspective thoracic surgery clinics 101016jthorsurg200707011 173 387393 online publication date 1aug2007 hesse b morise pothier c blackstone e lauer 2005 reliably predict longterm mortality exercise testing external validation american heart journal 101016jahj200409046 1502 307314 online publication date 1aug2005 hadjianastassiou v tekkis p poloniecki j gavalas goldhill 2004 surgical mortality score risk management tool auditing surgical performance world journal surgery 101007s0026800371746 282 193200 online publication date 1feb2004 shahian blackstone e edwards f grover f grunkemeier g naftel nashef nugent w peterson e 2004 cardiac surgery risk models position article annals thoracic surgery 101016jathoracsur200405054 785 18681877 online publication date 1nov2004 shahian normand torchiana lewis pastore j kuntz r dreyer p 2001 cardiac surgery report cards comprehensive review statistical critique11this review abridged version report submitted massachusetts cardiac care quality commission massachusetts legislature may 2001 annals thoracic surgery 101016s0003497501032222 726 21552168 online publication date 1dec2001 glance l osler 2001 beyond major trauma outcome study benchmarking performance using national contemporary populationbased trauma registry journal trauma injury infection critical care 1010970000537320011000000017 514 725727 online publication date 1oct2001 pitknen niskanen rehnberg hippelinen hynynen 2000 intrainstitutional prediction outcome cardiac surgery comparison locally derived model euroscore european journal cardiothoracic surgery 101016s1010794000005790 186 703710 online publication date 1dec2000 online publication date 1dec2000 attergrim j sterner claeson dharap gupta khajanchi kumar v gerdin wrnberg fox e 2018 predicting mortality international classification disease injury severity score using survival risk ratios derived indian trauma population cohort study plos one 101371journalpone0199754 136 e0199754 april 27 1999vol 99 issue 16 advertisement article informationmetrics copyright 1999 american heart associationhttpsdoiorg10116101cir99162067 originally publishedapril 27 1999 keywordsrisk factorseditorialsbypasspdf download advertisement