<title>Background</title> <p>Given that health care resources are scarce there is a need to evaluate the cost-effectiveness of different health care interventions. Within such studies economists generally seek to measure the benefits in terms of utility, a scale where 0 represents death and 1 is equivalent to full health, in order for the benefits of many interventions to be compared on a common scale [##UREF##0##1##, ####REF##15916116##2##, ##REF##16682976##3####16682976##3##]. However, as not all studies choose to measure outcomes in terms of utility, an increasing amount of research has now been conducted on mapping, where scores from a condition-specific (non preference-based) measure are 'converted' into a utility (preference-based) score using a pre-defined formulae [##UREF##1##4##]. Mapping thereby presents the possibility of estimating the cost-utility (i.e. the incremental cost per quality adjusted life year (QALY) [##UREF##0##1##]) of interventions that have previously only been evaluated using a condition-specific measure. Indeed a number of mapping models have now been developed [##REF##15319619##5##, ####REF##16893470##6##, ##REF##16495197##7##, ##REF##17532814##8##, ##REF##16855128##9##, ##REF##15449641##10##, ##REF##17665482##11##, ##REF##17645681##12##, ##REF##15090102##13##, ##REF##16193322##14##, ##REF##17697450##15##, ##REF##11310943##16##, ##REF##15185716##17##, ##REF##14583690##18##, ##REF##17343301##19####17343301##19##], the use of mapping has been considered by the UK National Institute of Health and Clinical Excellence (NICE) [##UREF##2##20##], and mapping has been used to estimate the utility scores, and in turn cost-effectiveness, of a number of health care interventions [##UREF##3##21##, ####REF##12890861##22##, ##REF##16938781##23####16938781##23##]. The role of this paper is to assess the criterion validity of such mapping procedures, as this represents an area where little research has been undertaken. We achieve this by comparing the actual estimated QALY gain associated with different interventions to the QALY gains predicted on the basis of mapping models, and similarly comparing the actual incremental cost per QALY estimates associated with those interventions to those derived from mapping models.</p>
<title>Methods</title> <title>Participants</title> <p>All individuals were taking part in the Lifestyle Interventions for Knee Pain (LIKP) study, which was designed to compare the effectiveness and cost-effectiveness of four different interventions. The four interventions were receipt of a leaflet, advice on knee strengthening exercises, dietary advice, and both dietary and exercise advice (hereafter these interventions are referred to as 1, 2, 3 and 4 as the main focus of this paper is methodological). Ethical approval for this study was granted by the UK Nottingham Research Ethics Committee. Recruitment into the LIKP study began in May 2003 and ended in March 2005, where all registered patients in five Nottingham general practices who were aged ≥ 45 years, and deemed (by their general practitioner) to be well enough to complete a questionnaire, were sent an ascertainment questionnaire. Additionally a local media campaign was conducted, which included adverts in the local press and on the local radio. Responding individuals were recruited into the LIKP study if they reported that they had had knee pain on most days of the last month, were aged ≥ 45 years, had a body mass index (BMI) &gt; 28.0 kg/m<sup>2</sup>, and gave consent to be randomised to one of the four interventions.</p> <title>Outcome measures</title> <p>At both pre-intervention (baseline) and post-intervention (at 6, 12 and 24 months) participants in the LIKP study were asked to complete both the WOMAC (Western Ontario and McMaster Universities Osteoarthritis Index) and the EQ-5D.</p> <p>The WOMAC contains 24 questions and measures the amount of pain (5 questions), stiffness (2 questions), and difficulty in physical functioning (17 questions), where the response options are none (0), mild (1), moderate (2), severe (3) or extreme (4) [##REF##3068365##24##]. Scores can thereby range between 0 and 20 on the pain sub-scale (pain), 0 and 8 on the stiffness sub-scale (stiffness), 0 and 68 on the functioning sub-scale (functioning), and sum to between 0 to 96 (total WOMAC), where higher scores denote a worse response [##REF##12465137##25##,##REF##15527668##26##]. Previous evidence of the adequate performance of the WOMAC has been shown for construct validity [##REF##12865040##27##] and responsiveness [##UREF##4##28##,##REF##15501407##29##].</p> <p>The EQ-5D has five questions, where the respondent is asked to report the level of problems they have (no problems, some/moderate problems, and severe/extreme problems) with regard to mobility, self-care, usual activities, pain/discomfort, and anxiety/depression [##REF##10158943##30##]. Responses to these five dimensions are converted into one of 243 different EQ-5D health state descriptions, which range between no problems on all five dimensions (11111) and severe/extreme problems on all five dimensions (33333). A utility score was assigned to each of these 243 health states using the York A1 tariff [##REF##9366889##31##], which was based on the preferences elicited from a survey of 3395 UK residents – EQ-5D scores range between -0.594 and 1 (full health).</p> <title>Statistical analyses</title> <title>Overview</title> <p>We adopted a split-sample approach to the mapping of condition-specific scores into utility scores. The baseline scores from the aforementioned LIKP study were used to develop various mapping models (to predict the EQ-5D scores). The performance of those models was then assessed on the post-intervention scores in order to identify our preferred model. Finally, for each of the four interventions, the actual QALY gains (over the 24 month trial period), and the incremental cost per QALY estimates, were compared to those that would have been predicted on the basis of our preferred model.</p> <title>Model specification</title> <p>In line with previous mapping models [##REF##16855128##9##,##REF##17665482##11##, ####REF##17645681##12##, ##REF##15090102##13##, ##REF##16193322##14####16193322##14##,##REF##11310943##16##, ####REF##15185716##17##, ##REF##14583690##18##, ##REF##17343301##19####17343301##19##,##REF##12890861##22##,##REF##16938781##23##], we used linear regression analysis to predict the relationship between scores on a condition-specific measure and scores on a utility measure. Using baseline WOMAC and EQ-5D data from the LIKP study, five models were developed, starting with the most parsimonious. In each of the models different baseline WOMAC scores took the form of independent variables and the baseline EQ-5D score acted as the dependent variable. The predictor variables in each of the five models were as follows.</p> <p>Model A: total WOMAC;</p> <p>Model B: pain, stiffness, functioning;</p> <p>Model C: total WOMAC, total WOMAC<sup>2</sup></p> <p>Model D: pain, stiffness, functioning, pain*stiffness, pain*functioning, stiffness*functioning, pain<sup>2</sup>, stiffness<sup>2</sup>, functioning<sup>2</sup>;</p> <p>Model E: best of above models plus patient characteristics of age and sex.</p> <title>Model performance</title> <p>We sought to identify our 'preferred' model, out of the five aforementioned models, by comparing actual EQ-5D scores to EQ-5D scores predicted on the basis of each of the five mapping models. This comparison was performed at 6, 12, and 24 months post-intervention for individuals who had complete study data (i.e. completed both the EQ-5D and each of the WOMAC sub-scales at all of the four time points within our study). Baseline data was not used within these comparisons as, in line with previous studies [##REF##17532814##8##,##REF##17343301##19##], we sought to assess the performance of the models on a different sample of data to that used to develop the models. We inferred the 'preferred' model to be the one with the lowest Mean Absolute Error (MAE), where the MAE was calculated by taking the average value of each absolute prediction error (the prediction error equals the difference between the actual EQ-5D score, for a particular individual, and, for the same individual, the EQ-5D score predicted on the basis of the mapping model). For each model we also report the adjusted r<sup>2</sup>squared and the root mean square error (RMSE) (the RMSE is the positive square root of the average squared prediction error). Finally, in order to assess how well the mapping formulae performs across the range of EQ-5D scores, we also plot the actual follow-up EQ-5D scores against the prediction errors (predicted score minus actual score) from our inferred preferred model.</p> <p>In line with the mapping models which are developed here, a previous study has also attempted to predict utility scores using scores on the WOMAC [##REF##17343301##19##]. The study differed from ours in that it measured utility using the Health Utilities Index [##REF##11802084##32##], rather than the EQ-5D, and whilst acknowledging that there is an argument that utilities derived from different instruments should not be compared [##REF##10982099##33##], we also sought to compare the utility scores predicted by the mapping models of Grootendorst et al. [##REF##17343301##19##] to our actual EQ-5D scores. Grootendorst et al. [##REF##17343301##19##] developed four models, but we were only able to predict utility scores using the coefficient values from two of their models (here referred to as G1 and G2) as the other two models used independent variables (e.g. duration of osteoarthritis) which were not available for the individuals within our study. Models G1 and G2 had the same independent variables as our Model D, and G2 also included the variables of age and sex. The performance of these two models was again assessed by calculating the MAE, RMSE and the adjusted r<sup>2</sup>squared. Additionally, we also identified our 'preferred' Grootendorst et al. model to be that which had the lowest MAE.</p> <title>Comparing actual trial results to those predicted using mapping models</title> <title>QALY gain</title> <p>The LIKP study sought to estimate the effectiveness of four different interventions. We thereby used the following methods to compare the mean QALY gain (measured from baseline over the 24 month trial period) for each of the four interventions, as estimated by actual data, to that predicted on the basis of our preferred mapping model and our preferred Grootendorst et al. model. With regard to actual data, for those participants who had complete study data, the baseline and post-intervention (6, 12 and 24 month) EQ-5D scores were used to estimate the QALY gain using the area under the curve (AUC) method, with adjustment for baseline scores [##REF##15497198##34##]. The mean QALY gains were then calculated by estimating the average QALY gain for the four groups of participants who received each of the four interventions.</p> <p>With regard to our preferred model, and our preferred Grootendorst et al. model, EQ-5D scores were predicted at baseline, 6, 12, and 24 months post-intervention for individuals who had complete study data (EQ-5D scores were predicted at all time points as this method would be used for studies which had not included a measure of utility). Using the same methods as for the actual EQ-5D data (see above), these predicted EQ-5D scores were then used to estimate the QALY gain for each individual who had complete study data, and the mean QALY gain for each intervention, where these calculations were performed for both our preferred model, and our preferred Grootendorst et al. model. Finally, we compared the mean QALY gain for each of the four interventions according to actual EQ-5D data to that predicted on the basis of our preferred mapping model, and our preferred Grootendorst et al. model. The paired t-test was also used to assess whether the actual mean QALY gains differed significantly (p &lt; 0.05) from those predicted on the basis of our preferred model and our preferred Grootendorst et al. model. Finally, it should be noted that discounting was not undertaken in any of the above analysis as we sought to identify the differences between the QALY gains based on actual EQ-5D scores and those based on predicted EQ-5D scores, and we sought to identify the difference that arose due to the use of the mapping procedure.</p> <title>Incremental cost per QALY</title> <p>As described elsewhere (Barton GR, Sach TH, Avery AJ, Doherty M, Jenkinson C, Muir KR. Lifestyle Interventions for Knee Pain: Cost-effectiveness analysis. Paper submitted for publication), levels of resource use were combined with unit cost data to estimate the change in cost over the two year study period for all participants in the LIKP study (costs were calculated at 2005/6 levels, but in order to ensure the same discount rate was applied to both costs and benefits future costs were not discounted). The change in costs, rather than total costs, was estimated due to the presence of baseline differences in the analgesic costs across each of the four interventions, and was calculated using the same aforementioned AUC technique as was undertaken for QALYs. The mean change in cost was thereby calculated for each of the four interventions. These values along with the previously calculated QALY gains based on i) actual EQ-5D scores, ii) the EQ-5D scores predicted from our preferred model, and iii) the EQ-5D scores predicted by our preferred Grootendorst et al. model were then used to estimate the incremental cost per QALY gain associated with each of the four interventions (this is commonly referred to as the incremental cost-effectiveness ratio (ICER) [##UREF##0##1##], and hereafter we refer to it as the incremental cost per QALY). Incremental cost per QALY estimates were made by ordering the four interventions from least costly to most costly, excluding those interventions which were dominated (had a higher mean change in cost and lower mean QALY gain than another intervention) or were subject to extended dominance (combinations of other interventions could provide a higher benefit at lower/equivalent cost), and then calculating the ICER (incremental cost/incremental effect) for remaining interventions. Separate incremental cost per QALY estimates were made using each of the aforementioned three different methods of calculating a QALY gain. Finally, it should be noted that all analyses were performed in either SPSS [##UREF##5##35##] or Microsoft Excel.</p>
<title>Results</title> <title>Participants</title> <p>Across the five general practices 12,500 individuals were sent an ascertainment questionnaire, and 8,044 (64.4%) were returned. Subsequently, 318 individuals met the entry criteria for the LIKP study, and gave consent to be randomised to one of the four interventions. An additional 71 individuals were recruited via the media campaign. The mean age of these 389 individuals was 62.0 years, 66.0% were female, and 23.4% were classified as overweight (BMI 25 to &lt; 30 kg/m<sup>2</sup>), 50.4% as class I obese (30 to &lt; 35 kg/m<sup>2</sup>), 16.9% as class II obese (35 to &lt; 40 kg/m<sup>2</sup>), and 9.9% as class III obese (≥ 40 kg/m<sup>2</sup>). At baseline 348 individuals fully completed both the EQ-5D and the WOMAC, and data for these individuals were used to develop the five mapping models. The mean score (95% confidence interval) for these 348 individuals was 0.557 (0.528 to 0.587) on the EQ-5D, 7.76 (7.39 to 8.13) on the pain sub-scale, 3.91 (3.74 to 4.07) on the stiffness scale, 27.89 (26.54 to 29.23) on the physical functioning scale, and 39.55 (37.77 to 41.34) on the total WOMAC scale.</p> <title>Statistical analyses</title> <title>Model Specification and performance</title> <p>The parameter estimates for each of the five models that we developed to predict the baseline EQ-5D scores are summarised in Table ##TAB##0##1##, where it should be remembered that a higher WOMAC score denotes a worse response. When these models were used to predict the EQ-5D scores for the 259 individuals who had complete study data, it can be seen that Model C had the lowest MAE (0.140) out of the first four models when the actual scores at 6, 12 and 24 months were compared to those predicted on the basis of these models. As such, Model E used the same independent variables as model C, with the additional variables of age and sex. Model E had an MAE of 0.129, and was thus deemed to be our preferred model (see Appendix 1 for full details of both Models C and E). By way of an example of how these models are used to estimate EQ-5D scores, our preferred model would predict that a male with the aforementioned mean baseline characteristics (age = 62 years; total WOMAC = 39.55) would have an EQ-5D score of 0.577 (-0.3474012785 + (-0.0005977709*39.55) + (-0.0001081560*39.55<sup>2</sup>) + (0.0326027536*62) + (-0.0002352456*62<sup>2</sup>) + (0.0475889687*0)), the actual mean baseline EQ-5D score was 0.566 (95% confidence interval 0.532 to 0.600). Figure ##FIG##0##1## shows how the prediction errors (predicted score minus absolute score) of our preferred model (E) vary according to the actual EQ-5D scores (6, 12 and 24 month post-intervention data are plotted).</p> <p>The parameter estimates for models G1 and G2 are published in the appendix of the paper by Grootendorst et al. [##REF##17343301##19##]. In terms of performance, model G1 had a lower MAE (0.142) than model G2 (0.144) and though both these MAE were higher than that for Model E they were lower than that for Model D, which used the same WOMAC predictor variables.</p> <title>Comparing actual trial results to those predicted using mapping models</title> <title>QALY gain</title> <p>The WOMAC and EQ-5D were fully completed at baseline, 6, 12 and 24 months post-intervention by 259 individuals (66.6% of trial participants). Based on the actual EQ-5D scores for these individuals the mean QALY gain (over the 2 year trial period, with adjustment for baseline scores), for each of the four interventions, was estimated to be 0.089, 0.081, 0.120 and 0.149, respectively. In Table ##TAB##1##2## these values are compared to the mean QALY gains predicted by our preferred model (E), and our preferred Grootendorst et al. model (G1). It can be seen that, for each of the four interventions, the mean QALY gains derived from both these preferred models were consistently lower than the actual estimated mean QALY gains, though the results were not significantly different.</p> <title>Incremental cost per QALY</title> <p>Based on the responses for the 259 individuals who had complete study data the mean change in costs (standard deviation) for each of the four interventions (1–4) was estimated to be £7.75 (£122.66), £321.12 (£131.12), £832.85 (£171.39) and £792.24 (£248.19), respectively. These four mean change in costs were subsequently combined with the estimated mean QALY gains, based on both actual EQ-5D data and our preferred mapping models (as shown in Table ##TAB##1##2##), to give the incremental cost per QALY estimates which are reported in Table ##TAB##2##3##. Based on the actual EQ-5D scores intervention 4 had both a higher mean effect, and lower mean increase in cost, than intervention 3 – intervention 4 thereby dominated intervention 3, and for similar reasons intervention 1 dominated intervention 2. The incremental cost per QALY (ICER) for intervention 4 was thereby calculated by comparing it to intervention 1, and was estimated to be £13,154 ((792.24-7.72)/(0.149-0.089)).</p> <p>These results, differed from those that were obtained when the EQ-5D scores were predicted on the basis of our preferred model, where though intervention 3 was still dominated by intervention 4 intervention 2 was no longer dominated by intervention 1. Instead intervention 2 had an ICER of £6,036 (when compared to intervention 1), and the ICER for intervention 4 was estimated to be £6,068 (when compared to intervention 2) (see Table ##TAB##2##3##). Similarly, though intervention 3 was still estimated to be dominated by intervention 4 when model G1 was used to predict the EQ-5D scores the incremental cost per QALY results were again different from those based on the actual EQ-5D scores: intervention 2 was now estimated to be subject to extended dominance as combinations of interventions 1 and 4 could provide a higher benefit at lower cost (see [##REF##15556954##36##] for further information on how to determine when an intervention is subject to extended dominance), and the ICER for intervention 4 was estimated to be £6,345 (when compared to intervention 1) (see Table ##TAB##2##3##).</p>
<title>Discussion</title> <p>Within this paper we have shown how mapping models can be used to predict the QALY gain associated with different interventions, and in turn calculate the ICER associated with different interventions. When these predicted results are compared to actual results we found that our preferred model consistently underestimated the mean QALY gain associated with the four compared interventions. The ICER of each of the four interventions, based on actual data, also differed from that based on our preferred mapping model (see Table ##TAB##2##3##) – the most effective intervention (intervention 4) was estimated to be more cost-effective according to our preferred mapping model (ICER = £6,068), compared to when actual data was used (ICER = £13,154).</p> <p>Within this paper we also calculated the incremental cost per QALY estimates for the four interventions in the LIKP study using a previously published mapping model based on the HUI3. This approach is justified by the following example. In a previous study by Thomas et al. [##REF##15934131##37##] it was found that a home exercise programme for people with knee pain was more effective than no intervention, and more costly, but as effectiveness was only measured on the WOMAC one could not compare the cost-effectiveness of this new intervention to other health-care interventions, or the cost-effectiveness threshold. One possible way of estimating the cost-effectiveness of the home exercise programme would be to convert the WOMAC scores into utility scores using the mapping scores published by Grootendorst et al. [##REF##17343301##19##]. Interestingly, within this paper, our preferred Grootendorst et al. model had a lower MAE than was the case in the original data set within which it was developed (0.142 compared to 0.1645). Furthermore, the predicted incremental cost per QALY estimates based on our preferred Grootendorst et al. model, which was developed using the HUI3, were also numerically closer to the actual incremental cost per QALY estimates, than was the case for the predicted cost per QALY estimates based on our preferred model, which was developed using the EQ-5D (see Table ##TAB##2##3##).</p> <title>Explanations</title> <p>One possible explanation for the above QALY differences is as follows. Figure ##FIG##0##1## shows that that the prediction errors of our preferred model tend to be increasingly positive for lower EQ-5D scores and increasingly negative for higher EQ-5D scores. This suggests that the regression would tend to over predict the EQ-5D score for those at low levels of utility, and under predict the EQ-5D score for those at high levels of utility. As the EQ-5D scores tend to increase post-intervention (baseline mean EQ-5D score = 0.566 and 24 month mean = 0.639, for N = 259), the consequence of this is that the final EQ-5D scores tend to be underestimated and thus the QALY gain associated with each of the four interventions also tends to be underestimated. This is further demonstrated by plotting the predicted errors against the actual EQ-5D scores at baseline (Figure ##FIG##1##2a##) and at 24 months post-intervention (Figure ##FIG##1##2b##), where the fact that the prediction errors are more likely to be negative at 24 months indicates that the actual scores were more likely to exceed the predicted scores at this time period, than was the case at baseline. However, it should be noted that, even though the effectiveness of each intervention was predicted to be lower by the mapping models, the most effective intervention (4) was estimated to have a more favourable incremental cost per QALY estimate according to the mapping models as the predicted mean QALY gain for this intervention was closer to the actual estimate, than was the case for the three other interventions (see Table ##TAB##1##2##).</p> <p>A further explanation as to why the QALY gains predicted by the mapping formulae tend to underestimate the actual QALY gain may be that the benefits of the interventions are not fully captured by the WOMAC, which concentrates on pain, stiffness and physical functioning. Thus, if there are other benefits, which are not detected by the WOMAC, then this may also explain why the mapping formulae tends to under predict the actual QALY gains associated with each of the four interventions. This hypothesis concurs with that of others who have pointed out that mapping models can only encompass the gains that are detected by the condition-specific measure [##REF##15449641##10##].</p> <title>Comparisons with other studies</title> <p>We are aware of two papers which have compared both the QALY gain and incremental cost per QALY predicted on the basis of different mapping formulae [##REF##15748287##38##,##REF##17493521##39##]. Pickard et al. [##REF##15748287##38##] used ten different mapping models, four of which were based on the SF-12 [##REF##8628042##40##], and six on SF-36 [##REF##1593914##41##]. They then used both the before and after scores from both a group of asthma patients, and a group of stroke patients, to estimate the 1 year QALY gain, and the associated incremental cost per QALY (for both groups of patients it was assumed that the incremental cost of the intervention was $2000 greater than standard treatment). They found that the QALY gain (incremental cost per QALY) estimates for the asthma patients ranged between 0.032 and 0.065 ($30,769 to $63,492), and that the QALY gain for the stroke patients ranged between 0.028 to 0.072 ($27,972 to $72,727) [##REF##15748287##38##]. Thus, the results of Pickard et al. [##REF##15748287##38##], are in line with ours, in that the estimated ICER varies according to which algorithm is used. This was also the case in the study by Marra et al. [##REF##17493521##39##], who used mapping to estimate the ICER associated with two different drug strategies for patients with rheumatoid arthritis. Using a previous database, mapping models were created by estimating the relationship between the Health Assessment Questionnaire [##REF##7362664##42##] and the utility measures of the EQ-5D [##REF##10158943##30##], SF-6D [##REF##11939242##43##], HUI2 and HUI3 [##REF##11802084##32##]. The four created mapping models were then used to estimate the QALY gain associated with the two different drug strategies in a different data set, where the results for the two treatments were 3.33 and 4.67 for the EQ-5D mapping model, 3.79 to 4.69 for the SF-6D, 4.16 to 5.33 for the HUI2, and 1.73 to 3.68 for the HUI3. These values were used to estimate the incremental effect, and when accompanied by estimates of the incremental cost, the ICER was estimated to range between $32,018 (HUI3) and $69,826 (SF-6D).</p> <p>We also sought to compare our results to those of others who have used similar techniques to estimate the relationship between scores on a condition-specific measure and scores on a utility measure [##REF##16855128##9##,##REF##17665482##11##, ####REF##17645681##12##, ##REF##15090102##13##, ##REF##16193322##14####16193322##14##,##REF##11310943##16##, ####REF##15185716##17##, ##REF##14583690##18##, ##REF##17343301##19####17343301##19##,##REF##12890861##22##,##REF##16938781##23##]. Not all of these studies reported the mean absolute error (MAE) but reported values did include &lt; 0.13 [##REF##15185716##17##], 0.14 to 0.16 [##REF##15185716##17##,##REF##14583690##18##], 0.1628 [##REF##17343301##19##] and 0.19 [##REF##17645681##12##], all of which are generally comparable to the MAE of our preferred model (0.129). Within these other studies, and the mapping models that we developed (see Table ##TAB##0##1##), there was also a tendency for the agreement between the observed and predicted utility scores to improve as further socio-demographic variables were used to predict the variation in the utility scores (in an attempt to ensure that others can use the mapping formulae that we developed we only included the socio-demographic variables of age and sex in our preferred model as we considered that other variables would not be routinely recorded in other studies).</p> <title>Implications</title> <p>One possible implication of the results presented here is that, as the scores predicted on the basis of mapping differ from actual scores, utility estimates that are based on mapping models should not be seen as a substitute for actual utility measurement. As a consequence, prospective clinical trials should seek to measure outcomes with a utility measure, rather than using a condition specific measure and a mapping model to estimate the utility gain associated with an intervention. There may however still be a role for mapping in terms of estimating the utility score in previous studies which have only included a non-preference based outcome measure. For example, in the aforementioned study by Thomas et al. [##REF##15934131##37##], it was found that the new intervention was more costly and more effective (on the WOMAC), but without translating the scores on the WOMAC into utility scores we have no way of trying to estimate whether the additional benefits are worthwhile i.e. we can not determine whether it is cost-effective to provide the new intervention, or whether it would be more cost-effective to spend scarce health care resources elsewhere. Indeed a recent report by the UK NICE relied heavily on the use of mapping to estimate the cost-effectiveness of a number of interventions concerned with the care and management of osteoarthritis in adults [##UREF##6##44##]. Further justification for the use of mapping is also provided by the results of this study, in that had we only measured outcomes with the WOMAC, and used a previously published mapping model [##REF##17343301##19##] to estimate the QALY gains, and incremental cost per QALY, associated with each of the four interventions (as we did in Table ##TAB##2##3##) then we would have come to the same conclusion i.e. that intervention 4 was the most cost-effective intervention as it has an incremental cost per QALY which is less than the £30,000 per QALY cut-off which has been argued to represent the approximate cost-effectiveness threshold which has been used by NICE [##REF##17717337##45##, ####REF##16213624##46##, ##REF##16735341##47####16735341##47##] (NICE states that it operates to a threshold range of £20,000 to £30,000 per QALY [##UREF##7##48##]).</p> <title>Strengths and weaknesses</title> <p>We consider the main potential limitation of our study to be that the results may not be generalizable. This arises because we only used the WOMAC to predict the EQ-5D score, and other studies which use a different condition-specific measure, or utility measure, may find the actual study results are more similar to those predicted on the basis of the mapping models. Our results are however important as evidence as to the validity of mapping, in terms of how closely incremental cost per QALY estimates based on actual results align to those based on mapping, can only be provided by a series of converging results [##UREF##8##49##]. Moreover, in an attempt to increase the generalizability of our results we sought to develop mapping formulae using the technique of linear regression as this is the technique that is most commonly used within the literature [##REF##16855128##9##,##REF##17665482##11##, ####REF##17645681##12##, ##REF##15090102##13##, ##REF##16193322##14####16193322##14##,##REF##11310943##16##, ####REF##15185716##17##, ##REF##14583690##18##, ##REF##17343301##19####17343301##19##,##REF##12890861##22##,##REF##16938781##23##]. We do however appreciate that, as the utility scale is bounded, the technique of linear regression can result in biased and inconsistent estimates [##REF##11284209##50##,##UREF##9##51##]. As such it may be that other techniques such as Tobit regression, the censored least absolute deviations (CLAD) estimator [##REF##16855128##9##] and restricted maximum likelihood (REML) [##REF##17532814##8##] could result in a smaller MAE and better prediction.</p> <p>A further potential weakness of this paper is that we have used various approaches (e.g. a complete case analysis and discounting of future costs and benefits at 0%) which might not be undertaken in standard cost-effectiveness analysis. Techniques such as discounting and imputation [##UREF##0##1##] were not used as the main focus of this paper was methodological, and we sought to identify the true difference between actual and predicted scores that arose due to mapping. Additionally, we did not explore the potential for selection bias that might arise due to the recruitment of participants from two different sources (recruitment via local general practices compared to the local media campaign).</p> <p>The main strength of our paper is that this is one of the first studies to compare the actual QALY gain associated with particular interventions to the QALY gain that would have been predicted, for the same interventions, on the basis of mapping. Moreover, this comparison has been undertaken using both mapping models developed from both the LIKP study data, and from a previous paper [##REF##17343301##19##]. We are aware of two papers [##REF##15748287##38##,##REF##17493521##39##] who make similar incremental cost per QALY comparisons. We build upon the first of these papers [##REF##15748287##38##] by making incremental cost per QALY comparisons using mapping models that are based on a condition-specific measure, rather than the more generic measures of the SF-12 [##REF##8628042##40##] and SF-36 [##REF##1593914##41##], by using actual cost estimates for actual interventions, and by making comparisons with a different measure of utility (the EQ-5D). We similarly advance upon the second paper [##REF##17493521##39##] as they undertook incremental cost per QALY calculations using a number of mapping formulae, but did not include an actual measure of utility within their comparisons.</p>
<title>Conclusion</title> <p>We have shown how mapping can be used to estimate both the QALY gain, and incremental cost per QALY, associated with different interventions, and compared these predictions to actual results. In our study the mapping models developed from the WOMAC tended to underestimate the QALY gain associated with each of four interventions, compared to that which was derived from actual EQ-5D scores. Similarly, the incremental cost per QALY estimates based on the mapping models also differed from those based on actual data. This suggests that future trials should include a measure of utility, however mapping may still be useful in estimating the cost-effectiveness of interventions which have previously only been evaluated with a condition-specific measure.</p>
<p>This is an Open Access article distributed under the terms of the Creative Commons Attribution License (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/2.0"/>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</p> <title>Background</title> <p>Mapping has been used to convert scores from condition-specific measures into utility scores, and to produce estimates of cost-effectiveness. We sought to compare the QALY gains, and incremental cost per QALY estimates, predicted on the basis of mapping to those based on actual EQ-5D scores.</p> <title>Methods</title> <p>In order to compare 4 different interventions 389 individuals were asked to complete both the EQ-5D and the Western Ontartio and McMaster Universities Osteoarthritis Index (WOMAC) at baseline, 6, 12, and 24 months post-intervention. Using baseline data various mapping models were developed, where WOMAC scores were used to predict the EQ-5D scores. The performance of these models was tested by predicting the EQ-5D post-intervention scores. The preferred model (that with the lowest mean absolute error (MAE)) was used to predict the EQ-5D scores, at all time points, for individuals who had complete WOMAC and EQ-5D data. The mean QALY gain associated with each intervention was calculated, using both actual and predicted EQ-5D scores. These QALY gains, along with previously estimated changes in cost, were also used to estimate the actual and predicted incremental cost per QALY associated with each of the four interventions.</p> <title>Results</title> <p>The EQ-5D and the WOMAC were completed at baseline by 348 individuals, and at all time points by 259 individuals. The MAE in the preferred model was 0.129, and the mean QALY gains for each of the four interventions was predicted to be 0.006, 0.058, 0.058, and 0.136 respectively, compared to the actual mean QALY gains of 0.087, 0.081, 0.120, and 0.149. The most effective intervention was estimated to be associated with an incremental cost per QALY of £6,068, according to our preferred model, compared to £13,154 when actual data was used.</p> <title>Conclusion</title> <p>We found that actual QALY gains, and incremental cost per QALY estimates, differed from those predicted on the basis of mapping. This suggests that though mapping may be of value in predicting the cost-effectiveness of interventions which have not been evaluated using a utility measure, future studies should be encouraged to include a method of actual utility measurement.</p> <title>Trial registration</title> <p>Current Controlled Trials ISRCTN93206785</p>
<title>Appendix 1</title> <p>Model C: Predicted EQ-5D score = 0.746652555353163 + (0.000810215321934668* total WOMAC) + (-0.000119664323424435* total WOMAC<sup>2</sup>)</p> <p>Model E: Predicted EQ-5D score = -0.3474012785 + (-0.0005977709* total WOMAC) + (-0.0001081560* total WOMAC<sup>2</sup>) + (0.0326027536*age) + (-0.0002352456*age<sup>2</sup>) + (0.0475889687*sex))</p> <p>sex is equal to 1 if Female and 0 if male.</p> <title>Competing interests</title> <p>The authors declare that they have no competing interests.</p> <title>Authors' contributions</title> <p>GB and TS conceived the idea for the paper, undertook the analysis and drafted the paper. CJ, AA, MD, and KM assisted in the acquisition of data, interpretation of the analysis, and commented on drafts of the manuscript. All authors read and approved the final manuscript.</p>
<title>Acknowledgements</title> <p>We thank all participants who completed the Lifestyle Interventions for Knee Pain (LIKP) study questionnaire. The LIKP study was funded by the UK Arthritis Research Campaign (ARC) (grant number 13550). The funding body have not commented on the work presented in this manuscript. A previous version of this paper was presented at the UK Health Economics Study Group, University of East Anglia, January 2008. Finally, we also acknowledge the useful comments made by two anonymous referees.</p>
<fig position="float" id="F1"><label>Figure 1</label><caption><p>Comparison of the actual EQ-5D scores and the prediction errors of Model E.</p></caption></fig> <fig position="float" id="F2"><label>Figure 2</label><caption><p>Comparison of the actual EQ-5D scores and the prediction errors of Model E.</p></caption></fig>
<table-wrap position="float" id="T1"><label>Table 1</label><caption><p>Parameter estimates for the five models which were used to predict the baseline EQ-5D scores.</p></caption><table frame="hsides" rules="groups"><thead><tr><td/><td align="center" colspan="5">Model</td></tr><tr><td/><td align="center">A</td><td align="center">B</td><td align="center">C</td><td align="center">D</td><td align="center">E</td></tr></thead><tbody><tr><td align="left">Intercept</td><td align="center">0.900‡</td><td align="center">0.886‡</td><td align="center">0.747‡</td><td align="center">0.691‡</td><td align="center">-0.347</td></tr><tr><td align="left">total WOMAC</td><td align="center">-0.009‡</td><td/><td align="center">0.001</td><td/><td align="center">-0.001</td></tr><tr><td align="left">pain</td><td/><td align="center">-0.012</td><td/><td align="center">-0.009</td><td/></tr><tr><td align="left">stiffness</td><td/><td align="center">0.006</td><td/><td align="center">0.092†</td><td/></tr><tr><td align="left">functioning</td><td/><td align="center">-0.009‡</td><td/><td align="center">-0.005</td><td/></tr><tr><td align="left">pain* stiffness</td><td/><td/><td/><td align="center">-0.005</td><td/></tr><tr><td align="left">pain*functioning</td><td/><td/><td/><td align="center">-0.001</td><td/></tr><tr><td align="left">stiffness*functioning</td><td/><td/><td/><td align="center">0.001</td><td/></tr><tr><td align="left">pain<sup>2</sup></td><td/><td/><td/><td align="center">0.002</td><td/></tr><tr><td align="left">stiffness<sup>2</sup></td><td/><td/><td/><td align="center">-0.011</td><td/></tr><tr><td align="left">functioning<sup>2</sup></td><td/><td/><td/><td align="center">0.000</td><td/></tr><tr><td align="left">total WOMAC<sup>2</sup></td><td/><td/><td align="center">-0.000‡</td><td/><td align="center">-0.000†</td></tr><tr><td align="left">age</td><td/><td/><td/><td/><td align="center">0.033*</td></tr><tr><td align="left">age<sup>2</sup></td><td/><td/><td/><td/><td align="center">0.000</td></tr><tr><td align="left">sex (if Female)</td><td/><td/><td/><td/><td align="center">0.048</td></tr><tr><td align="left">Range of predicted scores</td><td align="center">0.086 to 0.900</td><td align="center">0.086 to 0.900</td><td align="center">-0.235 to 0.748</td><td align="center">-0.111 to 0.852</td><td align="center">-0.184 to 0.828</td></tr><tr><td align="left">MAE</td><td align="center">0.148</td><td align="center">0.147</td><td align="center">0.140</td><td align="center">0.146</td><td align="center">0.129</td></tr><tr><td align="left">Adjusted r<sup>2</sup></td><td align="center">0.275</td><td align="center">0.274</td><td align="center">0.296</td><td align="center">0.299</td><td align="center">0.313</td></tr><tr><td align="left">RMSE</td><td align="center">0.189</td><td align="center">0.187</td><td align="center">0.185</td><td align="center">0.190</td><td align="center">0.180</td></tr></tbody></table></table-wrap> <table-wrap position="float" id="T2"><label>Table 2</label><caption><p>Mean estimated QALY gains based on both actual data and mapping models.</p></caption><table frame="hsides" rules="groups"><thead><tr><td align="center">Intervention</td><td align="center">1</td><td align="center">2</td><td align="center">3</td><td align="center">4</td></tr></thead><tbody><tr><td align="left">Actual results</td><td align="center">0.089 (0.330)</td><td align="center">0.081 (0.428)</td><td align="center">0.120 (0.450)</td><td align="center">0.149 (0.352)</td></tr><tr><td align="left">Model E</td><td align="center">0.006 (0.162)</td><td align="center">0.058 (0.186)</td><td align="center">0.058 (0.158)</td><td align="center">0.136 (0.209)</td></tr><tr><td align="left">Model G1</td><td align="center">-0.009* (0.146)</td><td align="center">0.027 (0.166)</td><td align="center">0.062 (0.149)</td><td align="center">0.114 (0.186)</td></tr></tbody></table></table-wrap> <table-wrap position="float" id="T3"><label>Table 3</label><caption><p>Estimated incremental cost per QALY based on both actual data and mapping models.</p></caption><table frame="hsides" rules="groups"><thead><tr><td align="center">Intervention</td><td align="center">1</td><td align="center">2</td><td align="center">3</td><td align="center">4</td></tr></thead><tbody><tr><td align="left">Actual results</td><td align="center">N/A</td><td align="center">D by 1</td><td align="center">D by 4</td><td align="center">£13,154</td></tr><tr><td align="left">Model E</td><td align="center">N/A</td><td align="center">£6,036</td><td align="center">D by 4</td><td align="center">£6,086</td></tr><tr><td align="left">Model G1</td><td align="center">N/A</td><td align="center">Subject to ED</td><td align="center">D by 4</td><td align="center">£6,345</td></tr></tbody></table></table-wrap>
<table-wrap-foot><p>* = p &lt; 0.05, † = p &lt; 0.01, ‡ = p &lt; 0.001, MAE = Mean Absolute Error, RMSE = Root Mean Squared Error. See Appendix 1 for full details of Model C and Model E.</p></table-wrap-foot> <table-wrap-foot><p>Standard deviations are presented in brackets, * = p &lt; 0.05 according to the paired t-test</p></table-wrap-foot> <table-wrap-foot><p>N/A = Not applicable (this intervention is the least costly and least effective), ED = Extended dominance, D = Dominated</p></table-wrap-foot>
<graphic xlink:href="1477-7525-6-51-1"/> <graphic xlink:href="1477-7525-6-51-2"/>
[{"surname": ["Drummond", "Sculpher", "Torrance", "O'Brien", "Stoddart"], "given-names": ["MF", "MJ", "GW", "BJ", "GL"], "source": ["Methods for the Economic Evaluation of Health Care Programmes"], "year": ["2005"], "edition": ["3"], "publisher-name": ["New York: Oxford University Press"]}, {"surname": ["Brazier", "Ratcliffe", "Salomon", "Tsuchiya"], "given-names": ["JE", "J", "JA", "A"], "source": ["Measuring and Valuing Health Benefits for Economic Evaluation"], "year": ["2007"], "publisher-name": ["New York: Oxford University Press Inc."]}, {"collab": ["National Institute of Health and Clinical Excellence"], "source": ["Guide to the Methods of Technology Appraisal"], "year": ["2007"], "comment": ["(Draft for consultation)."]}, {"surname": ["Barton", "Jobanputra", "Wilson", "Bryan", "Burls"], "given-names": ["P", "P", "J", "S", "A"], "article-title": ["The use of modelling to evaluate new drugs for patients with a chronic condition: the case of antibodies against tumour necrosis factor in rheumatoid arthritis"], "source": ["Health Tech Assess"], "year": ["2004"], "volume": ["8"], "fpage": ["1"], "lpage": ["104"]}, {"surname": ["Davies", "Watson", "Bellamy"], "given-names": ["GM", "DJ", "N"], "article-title": ["Comparison of the responsiveness and relative effect size of the western Ontario and McMaster Universities Osteoarthritis Index and the short-form Medical Outcomes Study Survey in a randomized, clinical trial of osteoarthritis patients"], "source": ["Arthritis Rheum"], "year": ["1999"], "volume": ["12"], "fpage": ["172"], "lpage": ["179"], "pub-id": ["10.1002/1529-0131(199906)12:3<172::AID-ART4>3.0.CO;2-Y"]}, {"collab": ["Statistical Package for the Social Sciences (SPSS)"], "source": ["SPSS for Windows, release 130"], "publisher-name": ["Chicago, Illinois: SPSS Inc"]}, {"collab": ["National Institute of Clinical and Health Excellence (NICE)"], "source": ["NICE Guideline project \u2013 Osteoarthritis: the care and management of osteoarthritis in adults"], "year": ["2007"]}, {"collab": ["National Institute of Health and Clinical Excellence"], "source": ["Guide to the Methods of Technology Appraisal"], "year": ["2004"], "publisher-name": ["London, UK: Abba Litho Sales Limited"]}, {"surname": ["Streiner", "Norman"], "given-names": ["DL", "GR"], "source": ["Health Measurement Scales: a practical guide to their development and use"], "year": ["2003"], "edition": ["3"], "publisher-name": ["New York: Oxford University Press"]}, {"surname": ["Green"], "given-names": ["WH"], "source": ["Econometric analysis"], "year": ["2000"], "edition": ["4"], "publisher-name": ["NJ: Prentice-Hall"]}]
Health Qual Life Outcomes. 2008 Jul 14; 6:51