as health care costs rise for all payers , medicaid programs are faced with the dual challenges of providing appropriate health care for some of society 's most vulnerable members and containing costs .
direct controls , such as limiting eligibility for programs or access to needed services , are unlikely to produce net budgetary relief , since public funds ultimately may be tapped for costly urgent care for those who can not get treatment by other means .
targeting high - cost patients for enhanced care management is another approach to defending limited funds against increasing health care costs .
the care management approach seeks to engage consumers in self - management of chronic conditions , while supporting physician adherence to evidence - based care guidelines in an effort to reduce the demand for future high cost , preventable utilization .
cost savings are not guaranteed , but the potential to simultaneously reduce costs and improve health outcomes is appealing . as part of act 191 ,
vermont 's health care reform legislation , ovha , which manages their publicly - funded health insurance programs , established a pair of integrated programs to offer care management services to all chronically ill members in the state who meet qualifying criteria ( maxwell , 2007 ; office of vermont health access , 2007a ) .
in general , members must have been diagnosed with one or more of 11 chronic conditions : ( 1 ) asthma , ( 2 ) diabetes , ( 3 ) chronic obstructive pulmonary disease , ( 4 ) low back pain , ( 5 ) congestive heart failure , ( 6 ) ischemic heart disease , ( 7 ) rheumatoid arthritis , ( 8) hypertension , ( 9 ) disorders of lipid metabolism , ( 10 ) depression , and ( 11 ) chronic renal failure .
members with medicare or other major third party insurance are ineligible , as are those in pharmacy - only benefit programs and programs that are paid from non - medicaid funding sources .
the chronic care management program ( ccmp ) serves the majority of the state 's medicaid beneficiaries with one or more chronic conditions .
member - level interventions range from mailings of condition - specific self - management literature and telephone access to health coaches to the development of an individual care plan and face - to - face patient support , depending on the needs of the member ( office of vermont health access , 2008 ) .
the care coordination program ( ccp ) is an intensive case management program staffed by teams of nurses and social workers and funded to serve the top 5 to 10 percent of eligible members with chronic conditions .
one challenge for the ccp is predicting which members are at greatest risk for costly and preventable service utilization . in the initial phase of program development , cases were selected based on prior cost and health care utilization ( i.e. , emergency department and acute inpatient hospitalization ) .
systems were put in place to facilitate referrals from hospitals and emergency departments , and cases could also be referred from outside sources ( e.g. , primary care physicians ) ( office of vermont health access , 2007b ) .
adoption of these recruitment techniques was expedient , helping the program get off the ground .
however , as the ccp develops and matures , considerations of efficiency and equity become increasingly important .
although prior cost and utilization can help to predict future resource use , by themselves they are somewhat crude indicators in that they do not distinguish between transitory and chronic needs .
health risk predictive modeling is a methodological approach that uses clinical diagnostic information to predict future cost .
while there will always be room for referral systems in a program such as the ccp , basing initial case selection on health risk predictive modeling output enables the program to efficiently evaluate all eligible members .
it may help to identify at - risk patients who do not have effective physician advocates and provide timely assessment of risk before patients require emergency department or inpatient care .
predictive models can be customized for a particular insurer or medicaid program , implicitly incorporating the effects of benefits , practice patterns , and characteristics of the population served .
customization is costly and beyond the capacity of medicaid budgets in many states ; models developed in other states can be adopted at much lower cost . however , selecting a model can prove challenging , as the best model for the intended use generally is not obvious .
the purpose of this study was to help the state of vermont choose a predictive modeling software tool that would select members with one or more chronic illnesses who are most likely to need and benefit from participation in an intensive care management program .
as a small state without the population necessary to create a custom model , vermont needed to find the best model from among the readily available alternatives .
criteria for the selected tool included the ability to predict future costs with reasonable accuracy , cost effectiveness , and a clinically - meaningful condition classification system .
we used 2 years of historical claims data to evaluate the potential of the diagnosis - based health risk predictive modeling approach to select high cost cases for the ccp .
three models were chosen for testing based on their methodological attributes and history of use with medicaid populations : the cdps version 2.5 ( university of california , 2008 ) ; dcg risksmart version 2.2 ( dxcg , 2008 ) ; and acg - pm version 7.0 ( johns hopkins university , 2008 ) .
we examined and compared the potential of each pre - existing model to predict high cost medicaid members in vermont .
well - established tools are available to estimate current illness burden from administrative claims data and predict future overall resource use .
health risk predictive models use demographics , diagnoses , prior cost and service utilization , or some combination of these data to predict cost and stratify the population into low , medium , and high - risk cases relative to the average health risk of the population .
statistical techniques distinguish between diagnoses that are likely to be costly in the future versus those that are transitory in nature .
models developed using data mining techniques , such as neural networks or artificial intelligence , were not considered for this study .
such models typically perform well , but may be  overfit  to the development population , performing less well when applied to other datasets .
they are not methodologically transparent and often fail to produce clinically - meaningful findings ( hartnell and mackinnon , 2003 ; martin , rogal , and arnold , 2004 ) . models based solely on pharmacy data , such as medicaidrx ( gilmer et al . , 2001 ;
although these models have the advantage of utilizing timely pharmacy claims data , which are not subject to the long run out period common with other types of claims ( e.g. , inpatient facility claims ) , their clinical classifications are less precisely defined , by nature , and they perform less well overall than models that include diagnoses from claims ( zhao et al . , 2005 ) .
we have also excluded models that rely on data on prior health care service utilization and procedures ( e.g. , surgery ) , owing to data limitations , though such information would be expected to improve model performance .
we chose to focus on readily - available models developed using clinically - meaningful and transparent methods for medicaid populations and utilizing international classification of disease , 9 edition , clinical modification ( icd-9-cm ) diagnosis data from claims .
all three models chosen for comparison were developed for medicaid populations and are being used in one or more states currently ; none were developed with vermont data .
the acg case - mix system was developed at johns hopkins university using health care claims data from a combination of commercial managed care and medicaid populations ( smith and weiner , 1994 ; weiner et al . , 1996 ; weiner et al . , 1998 )
. the acg - pm uses demographics , diagnoses , and pharmacy expenditures to predict future cost .
first , patients are assigned to acg actuarial cells , which are mutually exclusive patient clusters defined by age , sex , and combinations of diagnoses .
resource utilization scores are assigned to each cluster based on the mean cost of patients in that cell in the development dataset . in the second stage ,
risk scores are created using linear regression of future cost as a function of acg resource utilization score plus indicators of frailty , hospital dominant conditions , specific chronic conditions , and optionally , prior pharmacy spending ( weiner , 2005 ) .
this two - stage method , which allows the model to give higher weight to certain conditions and includes some prior expenditure data , improves prediction compared with using acg resource utilization scores alone to forecast future resource use .
although acg cells contain broad groupings of disease , the software produces for each member a complete list of conditions , grouped into clinically - meaningful categories .
a version of the acg software is available without a fee to state medicaid agencies . for information on ordering : see http://www.acg.jhsph.edu/html/orderacgproducts_medicaid.htm#cons .
the cdps model was developed at the university of california , san diego , and is made available at http://cdps.ucsd.edu/ ( kronick and dreyfus , 1996 ; kronick et al .
the model was developed using linear regression under the assumption that the effects of different categories of diagnoses on costs are additive .
that is , an individual 's risk score generally increases with each additional separate condition identified .
the most recent version of the model was developed with medicaid claims from a group of mainly midwestern states ( california , colorado , georgia , michigan , missouri , ohio , and tennessee ) covering the period from the early to middle 1990s .
the cdps disease classification system focuses on chronic conditions ; a subset of icd-9-cm diagnosis codes is grouped into condition categories .
a potential advantage of cdps is that its classification system was developed specifically for medicaid populations , so particular emphasis was placed on classifying mental health and substance use disorders .
separate risk scores are produced for temporary assistance for needy families ( tanf ) and supplemental security income ( ssi ) beneficiaries .
the dcg hierarchical condition category ( hcc ) approach was developed by researchers from boston university who were part of the team that created the cms - hcc model to adjust medicare capitation payments ( pope et al . , 2004 ) .
dxcg , inc . offers an annual license arrangement for it 's risksmart software which assesses illness burden and predicts future cost using the dcg methodology ( http://www.dxcg.com/ ) . like cdps ,
the dcg model includes several disease interaction terms and child - specific weights for certain conditions .
dxcg 's medicaid model was developed using massachusetts data and was most recently updated in 2007 .
separate risk scores are available for medicaid fee - for - service and managed care applications .
nearly all icd-9-cm diagnoses are grouped into detailed sets of clinically meaningful disease and condition categories .
although the dcg classification system is sometimes criticized for its handling of behavioral health conditions , one recent study predicting mental health and substance abuse cost in a veterans affairs ' population found slightly better performance for dcg models compared with cdps ( sloan et al . , 2006 ) .
in recent research by winkelman and mehmud ( 2007 ) the dcg and acg - pm commercial prospective models were found to perform similarly in terms of r ( acg - pm : 18.7 percent ; acg - pm without prior pharmacy cost : 16.2 percent ; dcg : 17.4 percent ) and mean absolute prediction error ( mape ) as a percentage of total actual costs ( acg - pm : 85.6 percent ; acg - pm without prior pharmacy cost : 90.4 percent ; dcg : 88.0 percent ) . both models outperform cdps ' medicaid model with tanf cost weights ( r = 12.4 percent ; mape = 95.8 percent ) .
however , it is impossible to know whether these results , which used data from a commercially - insured study population and compared overall model performance , are applicable to vermont 's medicaid population and the task of selecting high cost cases .
we used 2 years of prior vermont medicaid claims data to test the ability of each model to correctly identify future high - cost members .
diagnoses recorded during state fiscal year ( sfy ) 2005 ( july 2004june 2005 ) were used to predict cost in sfy 2006 ( july 2005-june 2006 ) .
data for this study were supplied by the ovha , which granted the authors access to six distinct data sets linkable using unique member and claim identifiers : ( 1 ) professional claims ( e.g. , emergency room , office visits , and other outpatient services ) ; ( 2 ) institutional claims ( e.g. , hospital and nursing home utilization ) ; ( 3 ) pharmacy claims ; ( 4 ) eligibility data ( e.g. , demographic information , dates of eligibility , eligibility category , and type of coverage ) ; ( 5 ) third party insurance coverage data ; and ( 6 ) provider data . after excluding members with fewer than 30 days of medicaid eligibility and those with medicare or other major insurance or with pharmacy - only benefits , we were left with data on 126,616 members in sfy 2005 and 121,059 members in sfy 2006 , or approximately 80 percent of vermont 's medicaid population
keeping only those with at least 11 months of medicaid eligibility in both years leaves 59,384 members . of these
winkelman and mehmud ( 2007 ) compared predictive modeling tools using both the offered models with out - of - the - box cost weights , which reflect the relationship between illness burden and cost from the model development population , and customized models with cost weights that were recalibrated to the study 's commercially - insured population dataset .
the customized models were modified to include prior cost data ; all outperformed the offered models .
prospective r results for each of the three customized models were nearly indistinguishable ( acg - pm : 22.1 percent ; cdps : 21.2 percent ; and dcg : 22.9 percent ) . unfortunately , we were not able to recalibrate the three models to fit vermont medicaid data owing to the small population of members with chronic conditions .
model calibration is usually performed with populations in the hundreds of thousands to 1 million or more members . running each software product
produced a set of prospective risk scores for the study population using the model 's out - of - the - box cost weights .
since all three models were developed on beneficiaries from other states , risk scores were renormalized to the mean of the study population for ease of interpretation .
a score below 1.0 indicates that the individual is expected to be less costly than average .
a score in excess of 1.0 indicates a higher - than - average health care cost risk ( e.g. , a score of 2.0 means that the person is predicted to be twice as costly as the population average ) .
we stratified the population by risk score , assigning cases to the top 10 and top 1 percent risk groups for each model based on their relative risk scores .
we then examined the top percentiles of actual year 2 cost and evaluated each model in terms of its ability to correctly predict future high - cost members .
we computed the sensitivity of the model to correctly pick those who were high cost , the specificity of the model to correctly exclude those who were not high cost , and the positive predictive value of the case selection tool .
we computed the sensitivity of each model , or the ability to correctly identify high cost members , defined as the number of true positives ( cases selected based on year 1 data that turn out to be high cost in year 2 ) as a percentage of true positives plus false negatives ( members not selected in year 1 that turn out to be high cost in year 2 ; also known as type ii error ) . by definition , sensitivity declines with increasing thresholds of risk , from a lower extreme where all members are selected and sensitivity is 100 percent to the upper extreme where no members are selected and sensitivity is 0 . we were also interested in the specificity of the model , or the ability to correctly exclude low - cost members , defined as the number of true negatives ( members not selected in year 1 that are not high cost in year 2 ) as a percentage of true negatives plus false positives ( cases selected in year 1 that are not high cost in year 2 , also known as type i error ) .
sensitivity and specificity are traded off as progressively smaller groups of cases are selected . to evaluate the overall model performance on both tests
, we produced receiver operating characteristic ( roc ) curves ( figure 1 ) for each model .
the best model will simultaneously exhibit the highest sensitivity and specificity combinations , bowing out from the 45 null line .
model performance was evaluated by calculating the area under the roc curve for each model and comparing confidence intervals .
we also were interested in the positive predictive value ( ppv ) , or the proportion of cases selected in year 1 ( true + false positives ) that are high cost in year 2 ( true positives ) .
this ratio is of practical value to care management program planners since it indicates the expected proportion of actual high - cost members among all selected cases for a given model .
it can be used to determine the approximate number of appropriate cases that will be selected for care management for a given number of cases included in the intervention group or to determine the size of the intervention group needed to ensure that a certain number of appropriate cases will be identified . adapting the ppv test
, we classified all cases picked by each model into four outcome categories , ( 1 ) low - cost , ( 2 ) moderate - cost , ( 3 ) high - cost and ( 4 ) very high - cost , depending on actual cost in year 2 .
this provided more information on the ability of the model to select the most appropriate cases for care management .
we defined a low - cost pick as a case selected in year 1 that does not meet the 50 percentile of actual cost in year 2 ( $ 4,061 ) , since at that threshold , intervention costs may exceed potential savings .
a moderate - cost pick was one that ranged from the 50 to < 90 percentiles .
a high - cost pick was one that ranged from the 90 ( $ 24,597 ) to < 99 percentiles and a very high - cost pick met or exceeded the 99 percentile ( $ 96,332 ) .
we evaluated the distribution of cases picked in the top 10 percent risk group , since that corresponds roughly to the number of members that vermont budgeted to serve in its most intensive intervention , the ccp , and in the top 1 percent risk group , since care management programs in states serving larger populations typically are forced to be more selective owing to budget constraints .
next , we examined the percentage of all year 2 high - cost members found in the top 10 and top 1 percent risk groups selected using year 1 data .
as a final practical test , we examined the performance of each model in terms of selecting cases who were at risk of future hospitalization .
avoiding preventable hospitalizations is often a key goal of care management , and members at risk of hospitalization may be particularly suitable candidates for intervention .
we examined the percentage of cases selected by each model who had one or more inpatient admissions in year 2 and the percentage of all hospitalizations in the data that were accounted for by cases in the high risk groups .
table 1 shows sensitivity , specificity , and positive predictive value of each of the models evaluated at the 90 percentile of year 2 actual cost .
we find that the three models have similar performance on the sensitivity test at the lowest cutoff points , but model performance diverges as fewer cases are selected , with the acg - pm model performing best .
the corresponding roc curve suggests that the acg - pm model is best , followed by the dcg model ( figure 1 ) .
this is confirmed in table 2 , which shows that the 95 percent confidence interval ( ci ) for the acg - pm model excludes the 95 percent ci of the dcg model , which , in turn , excludes the 95 percent ci of the cdps model .
that is , the area under the acg - pm model 's roc curve is significantly larger than the area under the dcg model 's roc curve , which in turn is significantly larger area than the area under the cdps model 's roc curve .
areas under the roc curves for cdps and dcg both improve ( cdps : 0.73 ; dcg : 0.78 ) and the differences between all models become insignificant when evaluated at the top 1 percent of total year 2 cost
. the findings for ppv are less definitive . at the lower risk cutoff points , where many cases are selected and the rate of false positives is high for all models
, cdps performs best . at the highest cutoff points , where fewer cases are selected
, the acg - pm model has the highest ppv . since , in practice , care management programs must choose relatively small percentages of members to serve , the lower range of risk percentiles is not as relevant and the acg - pm model can be said to perform best overall .
differences between the models are less pronounced when evaluated at the top 1 percent of cost ( not shown ) , though the dcg model appears to perform slightly better than the other models at the highest risk cutoff .
taking the top 10 percent risk group ( table 3 ) , we find that the acg - pm model has the smallest percentage of low - cost picks ( 10.1 percent ) and cdps the highest ( 22.2 percent ) . the acg - pm model also has the highest percentage of moderate and high - cost picks , but the dcg model produces the largest percentage of very high - cost picks . with the top 1 percent risk group , we find that the acg - pm model again has the fewest low - cost picks and most moderate - cost picks and the dcg model has the highest number of high and very high - cost picks .
examining the percentage of all high - cost members that are found in the top 10 percent risk group ( figure 2 ) , we find that the acg - pm model selects the highest percentage of all year 2 high - cost members when evaluated at the 50 - 90 percentile cost thresholds .
the acg - pm and dcg models pick similar percentages at the 95 percentile , and the dcg model picks the highest percentage at the 99 percentile . when the top 1 percent risk group is considered ( figure 3 ) , the acg - pm and dcg models perform similarly up to the 90 percentile cost threshold , but the dcg model picks higher percentages of high - cost members at the 95 and 99 percentile cost thresholds . finally , we compared the ability of the models to identify members at risk of hospitalization
. not surprisingly , we found that the number of hospitalizations and total cost in year 2 are strongly correlated in our data ( pearson correlation coefficient = 0.4288 ; p<0.0001 ) .
nevertheless , comparing performance in terms of predicting hospitalizations helps to further differentiate the models .
table 4 shows the percentage of cases selected in each models ' high - risk groups ( top 10 and top 1 percent ) with hospitalizations in year 2 , the mean number of hospitalizations per person , and the percentage of all hospitalizations that were accounted for by cases in the top risk groups for each model . for both the top 10 and top 1 percent risk groups
, we find that the acg - pm model performs best , with the top 10 percent risk group accounting for 36 percent , and the top 1 percent risk group accounting for 7 percent , of all hospitalizations in the data .
the dcg model outperforms cdps for the top 10 percent risk group , with 27 versus 23 percent of all hospitalizations , but the cdps and dcg models perform equally well for the top 1 percent risk group , each containing 4 percent of all hospitalizations .
we find that the acg - pm outperforms the other models in terms of sensitivity , specificity , and positive predictive value at the 90 percentile cost threshold , but not at the 99 percentile .
first , we classified cases selected in the top 10 percent risk group based on year 1 data into low , moderate , high , and very high - cost picks , depending on actual cost observed in year 2 .
second , we determined the percentage of high - cost members found in the top 10 percent risk group for each model .
the results showed that the acg - pm model generally outperforms the other models through the 90 percentile of cost .
however , the dcg model produces the highest percentage of high - cost and very high - cost picks and finds the largest percentages of high - cost members when evaluated at the 99 cost percentile . finally , we compared the number of hospitalizations in the highest risk groups and found that the acg - pm model outperformed the other two models for both the top 10 and top 1 percent risk groups . putting these results together ,
we conclude that the acg - pm model is generally preferred . however , if the care management program were required to concentrate its efforts on the top 1 percent of members in terms of total expected future cost , the dcg model would be preferred .
regression - based models are more flexible and predict better than cell - based models ( rosen et al . , 2001 ) .
the performance of the dcg model indicates the strength of the dcg classification system and regression methodology to predict the highest - cost members . on the other hand , while the diagnosis - based approach has been found to do a better job of finding future high - cost members and those with manageable chronic illness than simply looking at prior cost , methods that combine information both on diagnoses and prior cost tend to do best ( ash et al . , 2001 ; zhao et al . , 2002
this is supported by our findings that the acg - pm model , which includes pharmacy expenditure data , tends to perform best overall .
the superior performance of the acg - pm model on this outcome may be explained by the inclusion of variables indicating the number of hospital dominant conditions , particular chronic conditions , and a frailty flag , since these indicators of future cost are also likely indicators of risk of future hospitalization .
our mandate for choosing a particular health risk predictive modeling technology for vermont included a combination of methodological , performance , and cost criteria .
the cdps model is available free to all and a version of the acg - pm model is made available without a fee to state medicaid agencies .
however , to the extent that the dcg model does a better job of selecting high - cost cases , it will be a more fiscally prudent choice over time .
by contrast , the acg and , particularly , the dcg software packages provide richly detailed clinical classification systems .
it was determined that if either cdps or acg - pm outperformed dcg , the choice would be clear .
if dcg was found to be better or more flexible , the choice of model would depend on the magnitude of the differences in performance , expected savings from care management , and the cost of the software license .
ultimately , the dcg model may be a cost - saving technology if found to be much more accurate in picking cases for care management .
based on our analyses and consideration of the circumstance that the ccp provides coverage for the top 10 percent of members , the state chose the acg - pm model .
one limitation of this study is that we are only able to test currently available technology .
the field of health risk predictive modeling is one that has expanded in recent years and we may expect new developments in the field , including both improvements to existing models and the entrance of new models , to continue to occur in the coming years .
another study limitation is the possibility that our specific findings would not hold for other state medicaid programs .
although health risk predictive models are generally robust , if a health system is sufficiently different from that of the development population , different results may be found .
nonetheless , our approach to the selection of a predictive modeling tool for case selection may be of general interest to other states facing the problem of selecting cases for care management and choosing among the available alternatives .
in particular , our finding that differences between the models are small when predicting the very highest cost members may be helpful to states facing budgetary constraints .
our findings indicate that although most of the cases selected by each model exceeded median cost in year 2 , none of the models is able to predict with great accuracy the highest cost members . at best , the health risk predictive models we tested are able to identify approximately one - half of the very highest cost members ( i.e. , those costing > $ 96,000 per year ) .
this performance is in line with what is typically seen with prospective models in terms of predicting cost .
a barrier to predictive accuracy in general is the volatile nature of serious injuries and illnesses that result in high cost health care encounters or episodes .
a potential barrier to accuracy in this instance in particular is the lack of a customized model fit on a sample of vermont 's medicaid population with chronic conditions . whether this level of performance is adequate for the purposes of a care management program
another question is whether the data produced by these models is sufficiently timely to meet program needs .
some evidence suggests that models using real - time hospital admissions data could produce substantial savings for medicaid programs ( billings and mijanovich , 2007 ) .
the currency of hospital data may improve with increased adoption of electronic medical records ; presently , many states experience a lag of several months before such data are reasonably complete .
future research will consider whether adoption of predictive modeling technology was an improvement over previous case selection methods which focused on prior cost and service utilization .
health risk predictive models offer opportunities for efficiency and equity , but lack the immediacy and individual detail  particularly information on social factors  that care managers may require .
 medicaid agencies are beginning to turn to care management to reduce costs and improve health care quality . 
 one challenge is selecting members at risk of costly , preventable service utilization . using claims data from the state of vermont 
 , we compare the ability of three pre - existing health risk predictive models to predict the top 10 percent of members with chronic conditions : chronic illness and disability payment system ( cdps ) , diagnostic cost groups ( dcg ) , and adjusted clinical groups predictive model ( acg - pm ) . 
 we find that the acg - pm model performs best . 
 however , for predicting the very highest - cost members ( e.g. , the 99th percentile ) , the dcg model is preferred . 
Introduction
Models
ACG-PM
CDPS
DCG
Methods
Results
Discussion