there are growing numbers of people with chronic conditions with a particularly rapid rise in the number with multiple care needs .
the complex needs of people with multiple chronic conditions require the development of delivery systems that bring together a range of professionals and skills from both the cure and care sectors to meet those needs . despite this
, service delivery has developed in ways that have tended to fragment care , both within and between sectors , through for example structural and financial barriers dividing providers at the primary / secondary care and at the health and social care interface ; distinct organizational and professional cultures ; and differences in terms of governance and accountability .
a substantial number of evaluations have been carried out of interventions designed to improve the integration or coordination of care .
a systematic review based on 21 reviews and 85 primary studies   showed that many of these initiatives were effective in improving care , though many fewer resulted in a reduction in healthcare costs ( table 1 ) .
one of the conclusions of this and other reviews is that the effectiveness of attempts to provide better integrated care is highly dependent on the context in which the intervention takes place .
interventions can not be seen separated from the context in which they are introduced , and this has been an important guiding principle in the evaluation described in this paper . in response to concerns that the needs of the aging population for well - integrated care were increasing , the uk department of health for england announced in 2008 that a number of  integrated care pilots would be established . healthcare purchasers and providers
there was no specification as to the form that such integration should take , or client groups who should receive the intervention .
there were over 100 applications , and after a two - stage selection process , the department of health selected 16 pilots .
the localities of selected pilots and the main focus of each are described in annex 1 .
a team from rand europe , ernst and young llp and the university of cambridge were appointed to carry out a three - year evaluation that was augmented by the inclusion in the evaluation of experts from the nuffield trust .
the evaluation aims to answer the following questions : what approaches to integration have been employed by the pilots ?
what approaches to integration work well and in what contexts ? who benefits from integration , in what ways , and with what consequences for equity ?
what resources are required to make integration succeed and how can these be efficiently used ? in delivering integrated care in the english national health service , what policies and practices are : most likely to deliver the intended outcome , most capable of being implemented and most acceptable to patients , users , clinicians , managers and the wider public .
a mixed methods approach was adopted including interviews with staff and patients , non - participant observation of meetings , structured written feedback from sites , questionnaires to service users and staff , and analysis of routinely collected hospital utilisation data for patients / users who had been recruited into the pilots .
our approach to understanding the context in which integration takes place is based on two classifications relating to structure and function . at the start of the evaluation
these classifications were deliberately general to avoid focusing too early on very specific approaches to integrated care .
we wanted to accommodate the fact that that the pilots themselves were still refining their approaches .
integration can be seen as occurring at three levels [ 4 , 5 ] micro - level integration activities .
these promote integration among individual practitioners within a single organisation ( e.g. between doctors and nurses in a primary care practice setting ) . meso - level integration activities .
these promote integration among practitioners working in different organisations ( e.g. between gps and specialists ) .
this might include co - location of services , which could occur with or without macro - level activities , such as pooled budgets .
these promote integration designed to facilitate organisation - to - organisation working , e.g. across different sectors .
examples of these are pooled budgets or joint budget holding between health and social care services , employment of care staff in a single organisation , or structural changes to facilitate work across two or more organisations .
integration can be classified   in terms of organisational integration , where organisations are brought together by mergers or by structural change .
service integration , where different clinical services or support / back - office functions are integrated .
this classification will guide our analysis of the data , and our testing of the various hypotheses which arose during the course of the study .
these included hypotheses that integrated care would lead to the development of new organisational structures to support integration , changes in staff roles , increased staff job satisfaction , fewer unscheduled emergency hospital admissions , reduced length of hospital stay , increased patient satisfaction , and reduced cost .
it should be noted that the evaluation was designed and funded prior to the appointment of the integrated care pilots , so these hypotheses were developed during the first six months of the evaluation as a result of detailed interaction with the sites .
this unusual research design allowed for the evaluation to be tailored to the aims of the sites which were not known at the time the evaluation team was appointed .
our approach to understanding the context in which integration takes place is based on two classifications relating to structure and function . at the start of the evaluation
these classifications were deliberately general to avoid focusing too early on very specific approaches to integrated care .
we wanted to accommodate the fact that that the pilots themselves were still refining their approaches .
integration can be seen as occurring at three levels [ 4 , 5 ] micro - level integration activities .
these promote integration among individual practitioners within a single organisation ( e.g. between doctors and nurses in a primary care practice setting ) . meso - level integration activities .
these promote integration among practitioners working in different organisations ( e.g. between gps and specialists ) .
this might include co - location of services , which could occur with or without macro - level activities , such as pooled budgets .
these promote integration designed to facilitate organisation - to - organisation working , e.g. across different sectors .
examples of these are pooled budgets or joint budget holding between health and social care services , employment of care staff in a single organisation , or structural changes to facilitate work across two or more organisations .
integration can be classified   in terms of organisational integration , where organisations are brought together by mergers or by structural change .
service integration , where different clinical services or support / back - office functions are integrated .
this classification will guide our analysis of the data , and our testing of the various hypotheses which arose during the course of the study .
these included hypotheses that integrated care would lead to the development of new organisational structures to support integration , changes in staff roles , increased staff job satisfaction , fewer unscheduled emergency hospital admissions , reduced length of hospital stay , increased patient satisfaction , and reduced cost .
it should be noted that the evaluation was designed and funded prior to the appointment of the integrated care pilots , so these hypotheses were developed during the first six months of the evaluation as a result of detailed interaction with the sites .
this unusual research design allowed for the evaluation to be tailored to the aims of the sites which were not known at the time the evaluation team was appointed .
the evaluation described here adopts the approach of the  embedded evaluator. the evaluation activities form a distinct strand within the integrated care pilot programme , helping to co - produce the successful delivery of the programme , rather than a completely separate study focused solely on contributing to the scientific understanding of integrated care .
however , it is equally important that the evaluation contributes to scientific understanding and that it generates valid and independent evidence to support decision - making in the future .
the approach combines systematically collecting and synthesizing evidence from across all the pilots together with a deeper investigation of a smaller number of pilots in order to gain more detailed understanding of the structures , processes , costs and outcomes of integration .
integrated care pilots use a variety of integrating activities ( ranging from influencing , creating incentives , sharing information , creating new information systems and so forth ) and have a variety of objectives ( including improving the effectiveness and efficiency of services , enhancing patient reported outcomes and delivering measurable health improvements . in this context
, the research approach is multi - method in order to understand both the activities pursued and the outcomes achieved . the evaluation is based on six approaches to data collection and analysis : systematic qualitative data collection from all sites ( through a  living document which is a semi - structured document completed regularly by each pilot site ) . in - depth case studies of six sites (  deep dives ) including interviews with staff and patients / service users and non - participant observation of meetings [ see section systematic qualitative data collection from all sites ( the  living document ) for more on why we decided to use case studies ] .
difference in differences analysis of data on hospital utilization comparing patients / service users enrolled in pilots with control data .
analysis of costs ( combining data from qualitative case studies and quantitative data on service utilization ) .
evaluation involves a number of activities leading to an exercise of judgement [ 7 , 8 ] . in evaluating the complex set of activities which broadly sit under the heading of 
integrated care pilots we also seek to arrive at judgements which are seen to be legitimate by the stakeholders involved .
this requirement for legitimacy is one of the many ways in which  pure research is distinct from evaluation .
this legitimacy potentially involves five steps ( similar to those identified by scriven ): understand from those delivering the pilots and from those funding the initiative the criteria they consider to be applicable .
assess the contribution made by the agency / activity in achieving these standards and outcomes .
these steps protect the evaluators from the accusation of being arbitrary or otherwise non - rational , but an important part of the logic of the evaluation is to develop a set of hypotheses based on the  theory of change offered up by the pilots themselves . implicitly or explicitly , many evaluations of complex interventions use a  theory of change approach .
these evaluations aim not only to understand the contribution made by a programme or activity to achieving outcomes , but also to interrogate evidence and communicate findings to support both learning and accountability .
our approach takes as its starting point the argument of weiss [ 11 , p. 6667 ] that :  the concept of grounding evaluation in theories of change takes for granted that social programmes are based on explicit or implicit theories about how and why the programme will work  the evaluation should surface those theories and lay them out in as fine detail as possible , identifying all the assumptions and sub - assumptions built into the programme . the evaluators then construct methods for data collection and analysis to track the unfolding assumptions .
the aim is to examine the extent to which programme theories hold  the evaluation should show which of the assumptions underlying the programme are best supported by the evidence .  in this sense ,  theories of change
is a guiding approach rather than a methodology , and its successful delivery requires harnessing a range of methodologies , such as those outlined elsewhere in this paper .
first the approach requires us to not only look at the outcomes of the programme but to pay equal attention to processes .
this contrasts with more classical evaluation approaches which tend to look at outcomes first and then to look for evidence to support attribution .
embedded evaluator where the evaluator works closely with policy makers , practitioners and end users to understand and elaborate a sometimes changing theory of change . without losing their independence
, successful evaluators will understand the world of the policy makers , practitioners and service users , including an understanding of what motivates their behaviour .
thirdly , the approach requires an ability to reconstruct and represent the sequence of events connecting actions to each other and how these contributed to the outcomes identified , reconstructing at least the sequence of events and statistical co - variations , but preferably also identifying the causal mechanisms at work .
fourthly , the approach is sensitive to the possibility that during the life of a programme or intervention , initial theories of change may alter in response to learning or to exogenous events and that the evaluation needs to capture these changing understandings and actions .
fifthly , it will also be sensitive to the fact that different and potentially conflicting theories of change might be simultaneously pursued within any one programme .
collectively , these precepts describe an interest not only in causal effects ( what happens when an independent variable changes ) but also causal mechanisms ( what connects causes to their effects ) ; not only what officials say they do but what the evidence shows they do ; and not only what contribution stories practitioners tell themselves and others but also what really contributes to benefit .
therefore , theory building and testing is an important part of the approach taken but it does not start with a priori theoretical claims or assumptions .
the living document involves semi - structured data collection from all integrated care pilots at approximately six - monthly intervals during the evaluation .
a lead person is designated in each site to collate responses in the living document , but in most cases , this individual draws on a variety of sources in collating responses , and there is an expectation that the views of a wide range of stakeholders will be represented in the completion of the document .
the data collected in the living document are organised into a series of broad questions : development of the pilot and background information .
. who is doing what ? identifying the main people and organisations involved , and their roles in implementing the pilot .
processes  identifying the intended processes , and processes which have been implemented so far .
a description of progress to date , an assessment of progress against plan , and an outline of what has facilitated / prevented progress . sustainability .
an assessment of how much difference is really being made by the pilot itself , in the context of other health policy initiatives which are taking place concurrently .
resource implications of the pilot . without attempting to provide a precise monetary value to the outcomes of the pilot , an assessment of the costs of the pilot , and whether benefits might have been achieved more easily in other ways .
after each round of data collection , data from the living document are analysed , and feedback is given in two ways . first , limited feedback is given to each site , including the opportunity to specify where more detailed information is needed in future rounds of living document completion .
second , the overall themes emerging from the living document are analysed , and these are fed back in a single document to all sites after each round of data collection .
this analysis also contributes to  learning events ( conferences and teleconferences to address different issues of relevance to pilots ) which are being run by the department of health throughout the pilot period , and subsequent rounds of the living document are adapted in light of feedback from the sites .
we selected a range of types of pilots for in - depth case study to reflect the range of approaches in the pilots and then select a sample from these reflecting the need for variety and site s ability to support a more detailed evaluation .
for the depth case studies in six sites , we will structure the evaluation using an approach that combines logic modeling with process mapping of the patient journey .
these methods will complement each other in creating a full picture of the integration pathways .
logic models   provide a brief summary of the key elements of an intervention ( or programme , or project ) and organize inputs , processes , outputs and outcomes systematically .
they facilitate a focus on the causal links in the chain connecting the allocation of resources to the intended outcomes . as such
, they are well suited to supporting an understanding the  theory of change underpinning the activity and simultaneously identifying the sorts of data that might support or weaken that theory
. this approach will provide both a way to describe and communicate the different interventions but also to provide a basis for what , causally , is happening .
it will provide the framework for understanding how the inputs of a pilot are related to its outcomes and impacts .
they are especially helpful in developing a shared understanding of a process between stakeholders and serve as a reference point for stakeholders in the initiative or programme .
process mapping the service user experience , by contrast , involves understanding the motivations , experiences and outcomes of the various interactions between the service user and the ( integrated ) service . using these case studies
, we will address the following questions : what approaches to integration have been employed by the pilots ?
this will provide a richer description of models than is possible in the overall national evaluation by exploring experiences , motivations , relationships , processes and costs in more detail .
this will generate data linking putative causes to observed effects i.e. understanding causal mechanisms . who benefits from integration and in what ways ( what definitions are there of 
success ) ? this will identify how benefits are distributed and with what implications for equality .
what resources are required to make integration succeed and how can these be efficiently used ?
this will identify the descriptive categories of costs , establishing their dimensions , estimating overall costs , and suggesting how generalisable these findings might be .
the living document will help identify what types of costs become apparent at various stages of development of a project .
how the development of integrated care is facilitated or impeded by other current policies , e.g. payment by results , practice - based commissioning etc . from these analyses ,
we aim to identify what policies and practices are most suitable ( i.e. fit for purpose and likely to deliver the intended outcome ) ; most feasible ( i.e. capable of being implemented given the existing architecture of delivery and accountability ) ; and most acceptable ( i.e. likely to generate the support of the people who use services , clinicians and other professionals , managers and the wider public ) .
there will be three key data collection methods : semi - structured interviews with professionals and patients / service users , documentary analysis , non - participant observation of meetings . the qualitative data collection in the deep dive sites
will also be used to collect data for the economic analysis ( see below ) .
interviews with staff will concentrate on the experience of delivering care , interactions with other professional groups and organizations within the integrated care pilot , and understanding of implications for the wider care system .
interviews with patients and users will focus on the patient / user journey and experience and its relationship to changes in the integrated care pilots . in analysing data on service utilisation
, we will focus principally on hospital admissions as a key variable , as many of the sites have a focus on reducing such admissions .
data will be taken from hospital episode statistics ( he s ) , both for outpatient referrals , accident and emergency attendances and inpatient stays .
these will enable analyses of changes in a number of measures of hospital use including overall rates of emergency admissions , admissions for  ambulatory sensitive conditions ( see annex 2 ) , and length of stay .
we derived the list of ambulatory sensitive conditions from ahrq   and purdy et al . .
information will be available for the individuals enrolled in any intervention , and also for the whole populations of general practices which are participating in the integrated care pilot .
the data will be at person level but anonymised so that the research team can not identify sensitive personal information or individual identities . the nhs information centre for health and social care will act a trusted third party to handle any confidential information and create the anonymised linked fields for use by the research team .
one of the key challenges in undertaking analyses of changes in hospital use for complex interventions is that individuals may be selected for an intervention because they have a high use of health services .
the problem is that any subsequent fall in utilisation in this group may simply be due to regression to the mean  that is people reverting to a normal level of use irrespective of the intervention .
one way round this is to use an approach that allows us to standardise for differences in the risk of future admissions .
first , we will assess the impact of the intervention on individuals enrolled in the integrated care pilots .
information on the prior patterns of diagnoses and hospital utilisation will be used to stratify cases according to the risk of admission .
the actual level of utilisation before and after the agreed starting point in each pilot will be compared . in this way
we will be able to track levels of hospital use for cohorts of people for 23 years before they became part of the pilot .
these control cases will be matched on a number of variables including risk of admission ( or other hospital use ) ; major diseases recorded ; history of hospital use ; and characteristics of the area of residence , such as levels of deprivation .
trends in hospital use within the groups of selected control cases will then be used as a test of observed differences in those enrolled in the integrated care pilots .
second , we will quantify the effect of the interventions on wider groups of patients ( e.g. practice populations ) by matching utilization data to that from to similar practice populations in national he s datasets .
the population level analysis will assess whether the intervention might not only have an impact on individual patients but also upon the wider population .
both of these approaches to analysis are required as there might be an impact of the interventions on individuals ( e.g. a reduction in admissions ) which could not be demonstrated in the wider population .
this might be because resources were simply redistributed between groups at equal risk of admission , or because the numbers enrolled in the pilots were too small to show an effect on the wider population .
sample size calculations suggest that few of the integrated care pilot sites will enroll sufficient numbers for data from individual sites to be analysed .
we therefore intend to pool data from sites which have similar aims and are providing broadly comparable interventions .
it is not possible to say which sites will provide data that can be pooled , as all sites are still developing their interventions . however , it looks likely , for example , that several sites will be using a form of case management of high - risk patients with the aim of reducing hospital admission , and we will be able to pool data from such sites .
data will also be analysed on primary and social care utilisation ( from patient questionnaires ) .
however , these data are being collected primarily for the economic analysis , as none of the sites has reduction in primary care utilisation as their main goal .
our analysis strategy is built around a generalized difference - of - differences regression approach at the person level .
regression models appropriate for each of the outcome measures ( e.g. emergency admissions ) will be developed .
these may be poisson models , negative binomial models , or gamma models as required by the form of the measure .
each individual will contribute one or more time periods to the dataset in both the pre- and post - intervention periods .
these models will use covariates including basic demographics and historical utilization to control for potential differences between the intervention and control cases .
person level random effects will also be included in the models to adjust standard errors for the repeated measures within person .
in addition to the traditional covariate adjustment in the difference - of - differences model we will use propensity score based methods . in a combined dataset of intervention cases and
non - intervention controls a propensity score model will be fitted that uses available covariates to predict intervention vs. control status .
we may also use the propensity scores to produce analysis weights which can be combined with covariate adjustment to support 
doubly robust estimation combined with difference - of - differences modeling will provide intervention effect estimates that control for both observed differences between intervention and control groups as well controlling for unobserved but fixed person characteristics .
the aim of identifying risk - matched controls and using propensity score analysis is to allow so far as possible for unmeasured patient and system effects and therefore to increase our ability to draw conclusions about likely cause and effect from what inevitably remains observational data .
we are conducting two surveys to assess the experience of service users in 11 of the 16 pilots .
the survey is being administered in autumn / winter 2009 and will be repeated on the same sample of service users in autumn 2010 .
the questionnaire was developed using the intended outcomes identified by pilot sites in their applications to join the scheme .
this identified a number of domains which were common to most pilots and were therefore included in the questionnaire .
frequency of certain critical events ( notes unavailable , test duplicated , wrong medication or wrong dose of medication prescribed , no follow - up arrangements after hospital discharge ) .
in addition , a question on service usage was included to contribute to the analysis of health service costs ( see below ) .
the questionnaire is available from the authors . in selecting items to represent these domains , we drew questions were possible from existing validated instruments .
in particular we drew a substantial number of questions from the english national gp patient survey which is currently sent annually to 5.5 million randomly sampled patients ( www.gp-patient.co.uk ) . by matching the socio - demographic and health questions to this survey also , we will be able to conduct a difference in differences analysis with individual control patients drawn from responders to the national survey . for five pilot sites it was not appropriate to collect patient information using this questionnaire because of the nature of the intervention and/or the population group targeted by the intervention ( for example
questionnaires are being sent to up to 500 service users in each site . where the site has identified more that 500 service users by autumn 2009 , a random sample of 500 will be taken . where fewer than 500 service users have been identified by march 2010
where a site is enrolling patients / service users sequentially during autumn 2009/spring 2010 , all patients receive a questionnaire until 500 have been enrolled .
those individuals who receive a questionnaire in autumn 2009/spring 2010 will receive a second questionnaire in autumn 2010 .
for all service users , the site identifies the start data of any intervention , so that we can determine whether questionnaires returned have been completed before or after the start of the intervention .
we are conducting two cross - sectional staff surveys within the 16 pilot sites , involving health and social care staff ( including community nurses , gps and social workers ) , in spring 2010 , and repeated in spring 2011 .
the staff questionnaire has substantial sections for free text to allow staff to describe their experience of the pilot in more detail , and these sections will be transcribed for qualitative analysis .
the questionnaire includes sections on : job changes since the introduction of the integrated care pilot .
the first are staffs who are closely involved in the development of the pilot ( e.g. employed by the pilot ) .
the additional 3545 will be sent to stratified random samples of practitioners whose work might be altered by the pilot 
the first is to estimate costs in order to provide decision - makers in the health and social care systems with a basis for understanding the categories and potential range of costs associated with the integrated care pilots .
this will provide a sense of how much the approach might cost if it were implemented elsewhere . an important part of this will be through data collection in the deep dive sites where we will use the logic model , process maps , key informant interviews , and documentary evidence to produce estimates of the costs of providing integrated care .
this will enable us to identify the categories of cost and the scale of resources required to deliver different models of integration .
we aim to produce a clear understanding of the main categories of cost ( staff by grade , equipment , building , travel etc . ) , the likely range of costs within each category , and subsequently estimate best , worst and most likely case scenarios .
we will also distinguish between  set - up costs and  running costs , although in a fluid , adaptive and improving system it may be difficult to draw this distinction . for both of these
we need to distinguish between the costs associated with participating in the dh programme ( including , for example , participation in events , reporting , contributing to the national evaluation ) and the costs solely required to deliver the integrated care programme .
we also aim to gather data through time that can show how costs have altered in response to actual service delivery or in order to overcome changing circumstances etc .
we propose only to look at costs internal to the health and social care system ( including private sector partners ) but we will be aware that costs could potentially be externalised onto service users and carers and we will ask service users and staff to comment on their sense of the types and magnitudes of these costs . across the integrated care pilots where we are collecting quantitative data on hospital utilisation , data on hospital admissions and length of stay will be costed using standard nhs costs , and included in the controlled difference in differences analysis described in above .
the evaluation described here adopts the approach of the  embedded evaluator. the evaluation activities form a distinct strand within the integrated care pilot programme , helping to co - produce the successful delivery of the programme , rather than a completely separate study focused solely on contributing to the scientific understanding of integrated care .
however , it is equally important that the evaluation contributes to scientific understanding and that it generates valid and independent evidence to support decision - making in the future .
the approach combines systematically collecting and synthesizing evidence from across all the pilots together with a deeper investigation of a smaller number of pilots in order to gain more detailed understanding of the structures , processes , costs and outcomes of integration .
integrated care pilots use a variety of integrating activities ( ranging from influencing , creating incentives , sharing information , creating new information systems and so forth ) and have a variety of objectives ( including improving the effectiveness and efficiency of services , enhancing patient reported outcomes and delivering measurable health improvements . in this context
, the research approach is multi - method in order to understand both the activities pursued and the outcomes achieved . the evaluation is based on six approaches to data collection and analysis : systematic qualitative data collection from all sites ( through a  living document which is a semi - structured document completed regularly by each pilot site ) . in - depth case studies of six sites (  deep dives ) including interviews with staff and patients / service users and non - participant observation of meetings [ see section systematic qualitative data collection from all sites ( the  living document ) for more on why we decided to use case studies ] .
difference in differences analysis of data on hospital utilization comparing patients / service users enrolled in pilots with control data .
analysis of costs ( combining data from qualitative case studies and quantitative data on service utilization ) .
evaluation involves a number of activities leading to an exercise of judgement [ 7 , 8 ] . in evaluating the complex set of activities which broadly sit under the heading of 
integrated care pilots we also seek to arrive at judgements which are seen to be legitimate by the stakeholders involved .
this requirement for legitimacy is one of the many ways in which  pure research is distinct from evaluation .
this legitimacy potentially involves five steps ( similar to those identified by scriven ): understand from those delivering the pilots and from those funding the initiative the criteria they consider to be applicable .
assess the contribution made by the agency / activity in achieving these standards and outcomes .
these steps protect the evaluators from the accusation of being arbitrary or otherwise non - rational , but an important part of the logic of the evaluation is to develop a set of hypotheses based on the  theory of change offered up by the pilots themselves . implicitly or explicitly , many evaluations of complex interventions use a  theory of change approach .
these evaluations aim not only to understand the contribution made by a programme or activity to achieving outcomes , but also to interrogate evidence and communicate findings to support both learning and accountability .
our approach takes as its starting point the argument of weiss [ 11 , p. 6667 ] that :  the concept of grounding evaluation in theories of change takes for granted that social programmes are based on explicit or implicit theories about how and why the programme will work  the evaluation should surface those theories and lay them out in as fine detail as possible , identifying all the assumptions and sub - assumptions built into the programme . the evaluators then construct methods for data collection and analysis to track the unfolding assumptions .
the aim is to examine the extent to which programme theories hold  the evaluation should show which of the assumptions underlying the programme are best supported by the evidence .  in this sense ,  theories of change
is a guiding approach rather than a methodology , and its successful delivery requires harnessing a range of methodologies , such as those outlined elsewhere in this paper .
first the approach requires us to not only look at the outcomes of the programme but to pay equal attention to processes .
this contrasts with more classical evaluation approaches which tend to look at outcomes first and then to look for evidence to support attribution .
embedded evaluator where the evaluator works closely with policy makers , practitioners and end users to understand and elaborate a sometimes changing theory of change . without losing their independence
, successful evaluators will understand the world of the policy makers , practitioners and service users , including an understanding of what motivates their behaviour .
thirdly , the approach requires an ability to reconstruct and represent the sequence of events connecting actions to each other and how these contributed to the outcomes identified , reconstructing at least the sequence of events and statistical co - variations , but preferably also identifying the causal mechanisms at work .
fourthly , the approach is sensitive to the possibility that during the life of a programme or intervention , initial theories of change may alter in response to learning or to exogenous events and that the evaluation needs to capture these changing understandings and actions .
fifthly , it will also be sensitive to the fact that different and potentially conflicting theories of change might be simultaneously pursued within any one programme .
collectively , these precepts describe an interest not only in causal effects ( what happens when an independent variable changes ) but also causal mechanisms ( what connects causes to their effects ) ; not only what officials say they do but what the evidence shows they do ; and not only what contribution stories practitioners tell themselves and others but also what really contributes to benefit .
therefore , theory building and testing is an important part of the approach taken but it does not start with a priori theoretical claims or assumptions .
the living document involves semi - structured data collection from all integrated care pilots at approximately six - monthly intervals during the evaluation .
a lead person is designated in each site to collate responses in the living document , but in most cases , this individual draws on a variety of sources in collating responses , and there is an expectation that the views of a wide range of stakeholders will be represented in the completion of the document .
the data collected in the living document are organised into a series of broad questions : development of the pilot and background information .
. who is doing what ? identifying the main people and organisations involved , and their roles in implementing the pilot .
a description of progress to date , an assessment of progress against plan , and an outline of what has facilitated / prevented progress . sustainability .
an assessment of how much difference is really being made by the pilot itself , in the context of other health policy initiatives which are taking place concurrently .
resource implications of the pilot . without attempting to provide a precise monetary value to the outcomes of the pilot , an assessment of the costs of the pilot , and whether benefits might have been achieved more easily in other ways .
after each round of data collection , data from the living document are analysed , and feedback is given in two ways .
first , limited feedback is given to each site , including the opportunity to specify where more detailed information is needed in future rounds of living document completion .
second , the overall themes emerging from the living document are analysed , and these are fed back in a single document to all sites after each round of data collection .
this analysis also contributes to  learning events ( conferences and teleconferences to address different issues of relevance to pilots ) which are being run by the department of health throughout the pilot period , and subsequent rounds of the living document
we selected a range of types of pilots for in - depth case study to reflect the range of approaches in the pilots and then select a sample from these reflecting the need for variety and site s ability to support a more detailed evaluation .
for the depth case studies in six sites , we will structure the evaluation using an approach that combines logic modeling with process mapping of the patient journey .
these methods will complement each other in creating a full picture of the integration pathways .
logic models   provide a brief summary of the key elements of an intervention ( or programme , or project ) and organize inputs , processes , outputs and outcomes systematically .
they facilitate a focus on the causal links in the chain connecting the allocation of resources to the intended outcomes .
as such , they are well suited to supporting an understanding the  theory of change underpinning the activity and simultaneously identifying the sorts of data that might support or weaken that theory . this approach will provide both a way to describe and communicate the different interventions but also to provide a basis for what , causally , is happening .
it will provide the framework for understanding how the inputs of a pilot are related to its outcomes and impacts .
they are especially helpful in developing a shared understanding of a process between stakeholders and serve as a reference point for stakeholders in the initiative or programme .
process mapping the service user experience , by contrast , involves understanding the motivations , experiences and outcomes of the various interactions between the service user and the ( integrated ) service . using these case studies
, we will address the following questions : what approaches to integration have been employed by the pilots ?
this will provide a richer description of models than is possible in the overall national evaluation by exploring experiences , motivations , relationships , processes and costs in more detail .
this will generate data linking putative causes to observed effects i.e. understanding causal mechanisms . who benefits from integration and in what ways ( what definitions are there of 
success ) ? this will identify how benefits are distributed and with what implications for equality .
what resources are required to make integration succeed and how can these be efficiently used ?
this will identify the descriptive categories of costs , establishing their dimensions , estimating overall costs , and suggesting how generalisable these findings might be .
the living document will help identify what types of costs become apparent at various stages of development of a project . how the development of integrated care is facilitated or impeded by other current policies , e.g. payment by results , practice - based commissioning etc . from these analyses , we aim to identify what policies and practices are most suitable ( i.e. fit for purpose and likely to deliver the intended outcome ) ; most feasible ( i.e. capable of being implemented given the existing architecture of delivery and accountability ) ; and most acceptable ( i.e. likely to generate the support of the people who use services , clinicians and other professionals , managers and the wider public ) .
there will be three key data collection methods : semi - structured interviews with professionals and patients / service users , documentary analysis , non - participant observation of meetings . the qualitative data collection in the deep dive sites
will also be used to collect data for the economic analysis ( see below ) .
interviews with staff will concentrate on the experience of delivering care , interactions with other professional groups and organizations within the integrated care pilot , and understanding of implications for the wider care system .
interviews with patients and users will focus on the patient / user journey and experience and its relationship to changes in the integrated care pilots .
in analysing data on service utilisation , we will focus principally on hospital admissions as a key variable , as many of the sites have a focus on reducing such admissions .
data will be taken from hospital episode statistics ( he s ) , both for outpatient referrals , accident and emergency attendances and inpatient stays .
these will enable analyses of changes in a number of measures of hospital use including overall rates of emergency admissions , admissions for  ambulatory sensitive conditions ( see annex 2 ) , and length of stay .
we derived the list of ambulatory sensitive conditions from ahrq   and purdy et al . .
information will be available for the individuals enrolled in any intervention , and also for the whole populations of general practices which are participating in the integrated care pilot .
the data will be at person level but anonymised so that the research team can not identify sensitive personal information or individual identities .
the nhs information centre for health and social care will act a trusted third party to handle any confidential information and create the anonymised linked fields for use by the research team .
one of the key challenges in undertaking analyses of changes in hospital use for complex interventions is that individuals may be selected for an intervention because they have a high use of health services .
the problem is that any subsequent fall in utilisation in this group may simply be due to regression to the mean  that is people reverting to a normal level of use irrespective of the intervention .
one way round this is to use an approach that allows us to standardise for differences in the risk of future admissions .
first , we will assess the impact of the intervention on individuals enrolled in the integrated care pilots .
information on the prior patterns of diagnoses and hospital utilisation will be used to stratify cases according to the risk of admission .
the actual level of utilisation before and after the agreed starting point in each pilot will be compared . in this way
we will be able to track levels of hospital use for cohorts of people for 23 years before they became part of the pilot .
these control cases will be matched on a number of variables including risk of admission ( or other hospital use ) ; major diseases recorded ; history of hospital use ; and characteristics of the area of residence , such as levels of deprivation .
trends in hospital use within the groups of selected control cases will then be used as a test of observed differences in those enrolled in the integrated care pilots .
second , we will quantify the effect of the interventions on wider groups of patients ( e.g. practice populations ) by matching utilization data to that from to similar practice populations in national he s datasets .
the population level analysis will assess whether the intervention might not only have an impact on individual patients but also upon the wider population .
both of these approaches to analysis are required as there might be an impact of the interventions on individuals ( e.g. a reduction in admissions ) which could not be demonstrated in the wider population .
this might be because resources were simply redistributed between groups at equal risk of admission , or because the numbers enrolled in the pilots were too small to show an effect on the wider population .
sample size calculations suggest that few of the integrated care pilot sites will enroll sufficient numbers for data from individual sites to be analysed .
we therefore intend to pool data from sites which have similar aims and are providing broadly comparable interventions .
it is not possible to say which sites will provide data that can be pooled , as all sites are still developing their interventions .
however , it looks likely , for example , that several sites will be using a form of case management of high - risk patients with the aim of reducing hospital admission , and we will be able to pool data from such sites . data will also be analysed on primary and social care utilisation ( from patient questionnaires ) . however , these data are being collected primarily for the economic analysis , as none of the sites has reduction in primary care utilisation as their main goal .
our analysis strategy is built around a generalized difference - of - differences regression approach at the person level .
regression models appropriate for each of the outcome measures ( e.g. emergency admissions ) will be developed .
these may be poisson models , negative binomial models , or gamma models as required by the form of the measure .
each individual will contribute one or more time periods to the dataset in both the pre- and post - intervention periods .
these models will use covariates including basic demographics and historical utilization to control for potential differences between the intervention and control cases .
person level random effects will also be included in the models to adjust standard errors for the repeated measures within person .
in addition to the traditional covariate adjustment in the difference - of - differences model we will use propensity score based methods . in a combined dataset of intervention cases and
non - intervention controls a propensity score model will be fitted that uses available covariates to predict intervention vs. control status .
we may also use the propensity scores to produce analysis weights which can be combined with covariate adjustment to support 
doubly robust estimation combined with difference - of - differences modeling will provide intervention effect estimates that control for both observed differences between intervention and control groups as well controlling for unobserved but fixed person characteristics .
the aim of identifying risk - matched controls and using propensity score analysis is to allow so far as possible for unmeasured patient and system effects and therefore to increase our ability to draw conclusions about likely cause and effect from what inevitably remains observational data .
we are conducting two surveys to assess the experience of service users in 11 of the 16 pilots .
the survey is being administered in autumn / winter 2009 and will be repeated on the same sample of service users in autumn 2010 .
the questionnaire was developed using the intended outcomes identified by pilot sites in their applications to join the scheme .
this identified a number of domains which were common to most pilots and were therefore included in the questionnaire .
frequency of certain critical events ( notes unavailable , test duplicated , wrong medication or wrong dose of medication prescribed , no follow - up arrangements after hospital discharge ) .
in addition , a question on service usage was included to contribute to the analysis of health service costs ( see below ) .
the questionnaire is available from the authors . in selecting items to represent these domains , we drew questions were possible from existing validated instruments .
in particular we drew a substantial number of questions from the english national gp patient survey which is currently sent annually to 5.5 million randomly sampled patients ( www.gp-patient.co.uk ) . by matching the socio - demographic and health questions to this survey also , we will be able to conduct a difference in differences analysis with individual control patients drawn from responders to the national survey . for five pilot sites it was not appropriate to collect patient information using this questionnaire because of the nature of the intervention and/or the population group targeted by the intervention ( for example
questionnaires are being sent to up to 500 service users in each site . where the site has identified more that 500 service users by autumn 2009 , a random sample of 500 will be taken . where fewer than 500 service users have been identified by march 2010
where a site is enrolling patients / service users sequentially during autumn 2009/spring 2010 , all patients receive a questionnaire until 500 have been enrolled .
those individuals who receive a questionnaire in autumn 2009/spring 2010 will receive a second questionnaire in autumn 2010 .
for all service users , the site identifies the start data of any intervention , so that we can determine whether questionnaires returned have been completed before or after the start of the intervention .
we are conducting two cross - sectional staff surveys within the 16 pilot sites , involving health and social care staff ( including community nurses , gps and social workers ) , in spring 2010 , and repeated in spring 2011 .
the staff questionnaire has substantial sections for free text to allow staff to describe their experience of the pilot in more detail , and these sections will be transcribed for qualitative analysis .
the questionnaire includes sections on : job changes since the introduction of the integrated care pilot . perceived changes to the care that patients / service users receive .
the first are staffs who are closely involved in the development of the pilot ( e.g. employed by the pilot ) .
the additional 3545 will be sent to stratified random samples of practitioners whose work might be altered by the pilot 
the first is to estimate costs in order to provide decision - makers in the health and social care systems with a basis for understanding the categories and potential range of costs associated with the integrated care pilots .
this will provide a sense of how much the approach might cost if it were implemented elsewhere . an important part of this will be through data collection in the deep dive sites where we will use the logic model , process maps , key informant interviews , and documentary evidence to produce estimates of the costs of providing integrated care .
this will enable us to identify the categories of cost and the scale of resources required to deliver different models of integration .
we aim to produce a clear understanding of the main categories of cost ( staff by grade , equipment , building , travel etc . ) , the likely range of costs within each category , and subsequently estimate best , worst and most likely case scenarios .
we will also distinguish between  set - up costs and  running costs , although in a fluid , adaptive and improving system it may be difficult to draw this distinction .
for both of these we need to distinguish between the costs associated with participating in the dh programme ( including , for example , participation in events , reporting , contributing to the national evaluation ) and the costs solely required to deliver the integrated care programme .
we also aim to gather data through time that can show how costs have altered in response to actual service delivery or in order to overcome changing circumstances etc .
we propose only to look at costs internal to the health and social care system ( including private sector partners ) but we will be aware that costs could potentially be externalised onto service users and carers and we will ask service users and staff to comment on their sense of the types and magnitudes of these costs . across the integrated care pilots where we are collecting quantitative data on hospital utilisation , data on hospital admissions and length of stay will be costed using standard nhs costs , and included in the controlled difference in differences analysis described in above .
selecting evaluation frameworks always involves a degree of compromise to meet conflicting demands within a finite budget .
we have opted to balance the collection of data from across all the pilots with more detailed data from six deep dive sites .
we have also opted to focus the evaluation on what the pilots themselves told us they were seeking to achieve .
the benefit of this is that we will be in a position to provide an evaluation which is grounded firmly in what the pilots are seeking to do .
however , it also means that some theoretical propositions will be under - explored and that attention may be directed more towards intended outcomes than unintended outcomes .
however , we are satisfied that this risk is managed by independent data collection ( for example on service utilisation , service user surveys and staff surveys ) and by the iterative way of working with those responsible for running each project . in planning to assess the evidence produced by this evaluation , we have been influenced by the principles of realistic evaluation   in which the mechanism ( the intervention ) acts in context to produce the outcome . if there is a single lesson from previous evaluations of attempts to integrate or coordinate care , it is that the context in which an intervention is introduced is crucially important to its success or failure .
so , in this evaluation , we have committed substantial resources to the qualitative evaluation , knowing that these analyses will be critical to interpreting the results of quantitative analyses .
our approach is to understand not only the  dose , frequency and effect but to identify the way pilots learn , respond and evolve and to take into account the expectations and motivations of staff and patients to understand how complex and evolving projects might have lessons for others seeking to do related things in different contexts .
policy makers , professionals , managers , carers and patients are all part of an emergent process
. we do not expect to measure precisely all effects but we do expect to understand the likely scope of benefits and the scale of efforts required , to contribute to the analysis of health service interventions and so reduce decision - makers uncertainties about integrated care .
this evaluation will produce a portfolio of evidence including interviews , surveys , cost estimations and service utilisation data aimed at strengthening the evidence base for integrated care , and in particular identifying the context in which interventions are likely to be effective .
these data will support a series of evaluation judgements but it is important to recognise that they can not be arrived at by simple aggregation of data .
rather , the process locates the new data within the existing body of research and forms judgements about what is added and how compelling this additional evidence is , thus reducing uncertainties about the role of integrated care in improving the efficient and effective delivery of healthcare .
julius centre for patient oriented research , the university medical centre utrecht , the netherlands nick goodwin , phd , senior fellow , policy directorate , king 's fund , london , uk ingrid mur , dr .
( associate professor ) , department of health organisation , policy and economics ( h.o.p.e . ) , faculty of health sciences , university of maastricht , the netherlands
 backgroundin response to concerns that the needs of the aging population for well - integrated care were increasing , the english national health service ( nhs ) appointed 16 integrated care pilots following a national competition . 
 the pilots have a range of aims including development of new organisational structures to support integration , changes in staff roles , reducing unscheduled emergency hospital admissions , reduced length of hospital stay , increasing patient satisfaction , and reducing cost . 
 this paper describes the evaluation of the initiative which has been commissioned.study design and data collection methodsa mixed methods approach has been adopted including interviews with staff and patients , non - participant observation of meetings , structured written feedback from sites , questionnaires to patients and staff , and analysis of routinely collected hospital utilisation data for patients / service users . 
 the qualitative analysis aims to identify the approaches taken to integration by the sites , the benefits which result , the context in which benefits have resulted , and the mechanisms by which they occur.methods of analysisthe quantitative analysis adopts a  difference in differences approach comparing health care utilisation before and after the intervention with risk - matched controls . 
 the qualitative data analysis adopts a  theory of change approach in which we triangulate data from the quantitative analysis with qualitative data in order to describe causal effects ( what happens when an independent variable changes ) and causal mechanisms ( what connects causes to their effects ) . 
 an economic analysis will identify what incremental resources are required to make integration succeed and how they can be combined efficiently to produce better outcomes for patients.conclusionthis evaluation will produce a portfolio of evidence aimed at strengthening the evidence base for integrated care , and in particular identifying the context in which interventions are likely to be effective . 
 these data will support a series of evaluation judgements aimed at reducing uncertainties about the role of integrated care in improving the efficient and effective delivery of healthcare . 
Background
Analytical framework
Research methods
Principles guiding the evaluation
Systematic qualitative data collection from all sites (the Living Document)
In-depth case studies of six sites (Deep Dives)
Service utilisation
Questionnaires for service users
Questionnaires for health and social care staff
Economic analysis
Conclusion
Reviewers