Handrails through the Swamp? A Pilot to Test the Integration and Implementation Science Framework in Complex Real-World Research

: The socio-environmental challenges the world faces are ‘swamps’: situations that are messy, complex, and uncertain. The aim of this paper is to help disciplinary scientists navigate these swamps. To achieve this, the paper evaluates an integrative framework designed for researching complex real-world problems, the Integration and Implementation Science (i2S) framework. As a pilot study, we examine seven inter and transdisciplinary agri-environmental case studies against the concepts presented in the i2S framework, and we hypothesise that considering concepts in the i2S framework during the planning and delivery of agri-environmental research will increase the usefulness of the research for next users. We found that for the types of complex, real-world research done in the case studies, increasing attention to the i2S dimensions correlated with increased usefulness for the end users. We conclude that using the i2S framework could provide handrails for researchers, to help them navigate the swamps when engaging with the complexity of socio-environmental problems.

Schon [12] describes these great challenges as 'swamps'. In Schon's metaphor, these swamps are overlooked by disciplinary science, which exists on 'high ground' where "manageable problems lend themselves to solutions through research-based theory and technique" (p. 28).
The potential benefits of integrative research approaches have been canvassed by many: they achieve a better understanding and framing of the problem [13][14][15]; they provide new knowledge, insight, and innovations as a result of multiple disciplines, research approaches, or relevant knowledge sources coming together [4,5,[15][16][17]; they are more able to deal with complexity and uncertainty [18,19]; they produce more relevant, practical, legitimate, and socially robust knowledge [6,15,[20][21][22][23]; they can underpin informed policy and decision-making [4]; and they can achieve societal change [24]. Moral obligations for undertaking more integrated research have also been proposed: that those implicated in the science have a role in its production [25] and a universal drive towards greater participation [4].
In addition to these challenges, an overriding concern for disciplinary researchers when moving into the swamp is the discrepancy between the technical rigour of their high-ground disciplinary practice and the apparent sloppiness of the swamp [12], where research is structured not by an inquiry or a knowledge gap to be filled but by the task of solving real-world problems [22]. Research that acknowledges an intent to create change [7] requires researchers to shift from being producers of knowledge to being active contributors to a social process of tackling real-world problems [6]. Furthermore, complex systemic problems, such as climate change, are not neutral objects of inquiry [31] and the values of the individuals and institutions involved influence how the topic is researched. Working with complex, systemic problems requires different ways of researching [35]; the factors contributing to success in integrative research are unclear [13,36,37] and scientific practice trails aspirational research design for tackling complex problems [31]. This lag is compounded by the absence of comprehensive guidance on how to research complex real-world problems, which means newcomers often rely on intuition to invent ways of dealing with these challenges [6,11,12,38]. Even researchers with experience in integrative research often fail to systematically reflect on what worked well and what did not, resulting in unnecessary duplication with each new integrative research project [26].
In this paper, as a pilot we test the Integration and Implementation Sciences (i2S) framework [11], an integrative research framework that is targeted at complex real-world problems using seven agri-environmental case studies. We hypothesise that agri-environmental case studies in which concepts from the i2S framework are considered will be perceived as more useful by next users of the research. We assess the perceived usefulness by interviewing and surveying next users from each case study.
By evaluating a published framework, we aim to support those disciplinary scientists who have just arrived 'in the swamp' and who have limited capacity and time to seek and critique guidance on integrative research from fragmented and potentially inaccessible body of literature [11,22]. We also seek to augment the i2S framework as a potential set of handrails for researchers in the swamp, with practical examples of successful integrated research activities in agri-environmental research projects.
There are multiple approaches for tackling complex real-world problems, such as systems engineering [39], transdisciplinarity [40][41][42], systemic intervention [43,44], and co-innovation [45]. All of these approaches take a problem orientation, coming from the perspective of tackling the problem and not from a disciplinary approach, and they inherently recognise that there will be differing views on what the problem is. Similarly, all the approaches recognise that not only multiple disciplines but multiple sources of knowledge are important, that achieving change involves more than supplying a simple technical solution, and that working with stakeholders is key to ultimately embedding the desired impact. In addition, some approaches also specifically consider the behaviours of the team and how the team works together. However, a key reason for choosing the i2S framework for this pilot is the explicit way it deals with unknowns. In complex socioenvironmental problems, all decisions have to deal with uncertainty, and as the issues become more complex, the different dimensions of uncertainty become more apparent [46]. As decisions still need to be made, this means that an understanding of the inevitable uncertainty of the science is integral to the decisions being made [47]. In contrast to many research traditions where the unknown is seen only as the substrate that is converted into knowledge, the i2S signals a different approach to understanding and managing unknowns through bringing together different approaches to provide a rich understanding and ways of dealing with unknowns [11].

The Integration and Implementation Science Framework
The i2S framework is structured around three domains: synthesising disciplinary and stakeholder knowledge (bringing together what is known and researching a problem using both research and practical experience); understanding and managing unknowns (appreciating that everything about a complex problem cannot be known and unknowns need to be managed and accounted for); and providing integrated research support for policy and practice change (actively supporting the implementation of the research findings).
Five questions applicable across each domain Figure 1. These questions cover (1) what the research is aiming to achieve and who will benefit; (2) what knowledge is synthesised, which unknowns are considered, and which aspects of policy or practice change are targeted; (3) how the knowledge is synthesised, how diverse unknowns are understood and managed, and how integrated research support is provided; (4) what circumstances influenced the research; and (5) an evaluation of how well 1-4 were done [11].

Methods
To test the hypothesis that using the concepts in the i2S framework renders research more useful to next users, we test the i2S framework using use seven case studies. Figure  2 shows how the case studies were used to test the hypothesis.

Methods
To test the hypothesis that using the concepts in the i2S framework renders research more useful to next users, we test the i2S framework using use seven case studies. Figure 2 shows how the case studies were used to test the hypothesis.

Choosing Case Studies
A case study investigates real-life phenomena through an analysis of events, conditions, and the relationships between these [48]. In social science, case studies are used when trying to answer how and why questions [49]. In our research, multiple case studies were selected to understand how the i2s framework is perceived by next users.
Seven cases (see Table 1) were chosen through purposive theoretical sampling, meaning the cases were chosen for theoretical and not statistical reasons [50,51]. Our cases cover a range of agri-environmental problems: they were all considered complex, they all had multiple stakeholders, multiple relevant scientific disciplines, recognised that other sources of knowledge beyond scientific knowledge needed to be included and were characterised by there being a real-world problem; in other words, in each case there were people trying to tackle this problem through policy or practice change. There were also practical reasons for our case study selections, for example, we chose projects with which at least one of the authors was familiar or involved, thereby facilitating access to participants or information. Following the guidance of Yin [52], case study data collection used multiple rather than single sources of evidence to provide triangulation, a case study database was created to store information, and a clear chain of evidence was maintained.

Complexity Assessment
The case studies were all assessed for their degree of complexity [53]. A three-item measure for project complexity was used, developed from Arkesteijn et al.'s [54] model of complexity [53] and using three measures: scientific disagreement, stakeholder disagreement, and systemic stability (how resistant the system is to change). Each dimension was rated on a 12 point scale: the higher the score, the greater the contribution of that dimension to project problem complexity. Following Patton [55], the scale's levels of complexity was categorised as 1-3 = simple problem, 4-6 = a complicated problem, 7-9 = a complex problem, and 10-12 = a seriously complex problem [53]. Only case studies that scored 7 or over in at least two of the three dimensions were included in the research.

Choosing Case Studies
A case study investigates real-life phenomena through an analysis of events, conditions, and the relationships between these [48]. In social science, case studies are used when trying to answer how and why questions [49]. In our research, multiple case studies were selected to understand how the i2s framework is perceived by next users.
Seven cases (see Table 1) were chosen through purposive theoretical sampling, meaning the cases were chosen for theoretical and not statistical reasons [50,51]. Our cases cover a range of agri-environmental problems: they were all considered complex, they all had multiple stakeholders, multiple relevant scientific disciplines, recognised that other sources of knowledge beyond scientific knowledge needed to be included and were characterised by there being a real-world problem; in other words, in each case there were people trying to tackle this problem through policy or practice change. There were also practical reasons for our case study selections, for example, we chose projects with which at least one of the authors was familiar or involved, thereby facilitating access to participants or information. Following the guidance of Yin [52], case study data collection used multiple rather than single sources of evidence to provide triangulation, a case study database was created to store information, and a clear chain of evidence was maintained.

Complexity Assessment
The case studies were all assessed for their degree of complexity [53]. A three-item measure for project complexity was used, developed from Arkesteijn et al.'s [54] model of complexity [53] and using three measures: scientific disagreement, stakeholder disagreement, and systemic stability (how resistant the system is to change). Each dimension was rated on a 12 point scale: the higher the score, the greater the contribution of that dimension to project problem complexity. Following Patton [55], the scale's levels of complexity was categorised as 1-3 = simple problem, 4-6 = a complicated problem, 7-9 = a complex problem, and 10-12 = a seriously complex problem [53]. Only case studies that scored 7 or over in at least two of the three dimensions were included in the research. To support and inform a collaborative policy process in setting water quality and quantity limits in the Selwyn Waihora catchment. [47,56] 2

Matrix of Good Management
To define primary industry-agreed good management practices and model the nutrient losses from farms operating using good management practice.

Nutrient Management
To develop and test on farm practices to help farmers to comply with changing and increasingly stringent regional water quality regulations. [60]

Log Segregation
To develop cost-effective approaches to characterise and deal with variation in wood properties within and between trees to enhance value-added production. [45]

Heifer Rearing
To improve the reproductive performance of New Zealand's dairy herd by lifting the proportion of heifers entering the national herd at target live weight. [60] 6 Water Use Efficiency To improve on-farm irrigation decisions using better characterisation of irrigation demand and accurate short-term weather forecasts.

Deconstructing the i2S Framework
To systematically collect data on the case studies and permit us to score them, we first deconstructed the domains and questions of the i2S framework into discrete 'elements'. We did this for the first four questions in each domain. Initially, we identified 59 elements across the three domains. These elements became probe questions designed to elicit narrative conversation regarding the implementation of the identified i2S elements in a project. However, after piloting the probe questions on fellow researchers familiar with the project, we discarded several of the element/probes that not did elicit a clear response or that were repetitive of other probes/elements. Thus, 43 elements were identified across the three domains, with domain 1 containing 19, domain 2 containing 9, and domain three containing 15 (for more information on the operationalisation of the framework into elements, see [53,63]). Appendix A presents a table in which the three domains, the four questions in each domain, and the elements associated with each of the questions are listed. The fifth questions in each domain are evaluation questions and do not introduce new elements. To collect data on the fifth questions, the project teams associated with each case study were asked to evaluate their project in each domain, reflecting on the decisions they made, the methods they used, and the outcomes they achieved. The questions used were drawn from Bammer [11] (pp. 55, 99, 141) and further refined in discussion with Gabriele Bammer; they are included in Appendix A.

Case Study Data Collection
Researchers in each of the seven-case study project teams were invited to a day-long workshop (three workshops in total) where they worked with a facilitator who used the domains, questions, and element probes to elicit a description of each case study through the lens of the i2S framework. For case studies 3, 4, 5, and 6, project team members recorded the project descriptions and evaluations themselves. For case studies 1, 2 and 7, either one of the authors recorded the information (case studies 1 and 7) or the workshop was recorded and transcribed (case study 5). The workshops, and the subsequent process for deriving interview and survey questions, were approved by the Manaaki Whenua-Landcare Research social ethics process (1617/06).

Scoring the Case Studies
Translating the narrative project descriptions into quantitative scores, and assessing the degree to which each element of the i2S was implemented in the projects, was a subjective rating process, and potentially subject to individual bias. Therefore, we took the approach described below to reduce individual bias and to ensure that analysts were providing comparable evaluations (for further information about this process, see [53]). Three analysts independently assessed each case study description and drew out pertinent information from the qualitative data on how the case study addressed each of the 43 i2S elements and for the team self-evaluation. For each assessment, one of the analysts had some knowledge of the case study. Other analysts had experience of working with qualitative data. The analysts then rated the degree to which each element was addressed based on the evidence in the case study descriptions. They also rated the team self-evaluation of project performance to generate a score for the team's self-assessment. The analysts used a five-point scale with descriptive anchors at the scale mid-point and the two endpoints. For the assessment of the consideration of i2S elements, 0 = the element was absent or not addressed in the data describing the project; 2 = the element was present or addressed to a moderate degree; and 4 = the element was strongly present or addressed to a high degree. For the assessment of the team self-evaluation, 0 = very poor; 2 = adequate; and 4 = excellent. The three analysts then compared their ratings for each of the elements of each project and the team self-evaluation. If their independent ratings for individual elements differed by more than one scale point, analysts discussed and reconsidered the original qualitative data and reached agreement (within one scale point) for that element. The mean of the three analysts' ratings was taken to provide the final rating for each element. In total, seven analysts were used across the seven case study projects, including three of the authors.
For each case study, an average score was calculated for each of the three i2S domains, and an overall project average score was calculated from the average domain scores. In this way, each of the i2S domains was weighted equally.

Next-User Assessments
Case study project team members identified all of the immediate 'next users' of their research and their roles (e.g., decision-maker, planner, and industry representative). These next users were invited to participate in an evaluation of the case studies. Participants were asked about their experiences of the case study research projects and the usefulness of the research process and outputs for their (next user) purpose through interviews or an online survey ( Table 2). The interview (Appendix B) and survey questions (Appendix C) did not ask about each of the 43 i2S elements but related more broadly to one of the three i2S domains. A final section of the survey and interview asked for an overall project assessment. Interviews were conducted by the second and third authors. All interviews were recorded and transcribed, and the chosen analysts analysed the transcriptions. The survey was administered by the second author. For the interview data, three analysts independently assessed each interview transcription and drew out pertinent information from each interview question response. From this qualitative data, they rated the perceived usefulness of the research for each next user using a five-point scale with descriptive anchors at the scale mid-point and end-points where: 0 = the research was not at all useful; 2 = the research was moderately useful; and 4 = the research was very useful. The analysts compared their ratings for each interview question as before, reaching agreement (within one scale point) for each rating. The final rating for each interview question was the mean of the three analysts' ratings.
For the survey data, the next users rated the research processes and outcomes on a five-point scale, and an average of the next user scores was taken.
From both the interview and survey data, an average score of the usefulness of the research was calculated from the questions in each domain for each next user. All the next-user domain scores for each domain were pooled to calculate an average domain score, and an overall project score was calculated from the average domain scores.

Study Limitations
The limitations to this pilot study include the limited number of available next users of each of the case study projects. Due to the nature of the assessments, the next users needed to be close enough to the project to judge the usefulness of different dimensions of the project. This limited the pool of next users available to participate. For example, in one case there were only two next users. A further limitation of this pilot is the mixed approach to next-user assessments, with two cases using interviews and five cases using surveys. This was the result of time and resource constraints. The impacts were minimised as much as possible by designing the survey based on the interview questions.
These limitations would be addressed by expanding this pilot to more case studies. Table 3 and Figure 3 show the degree to which each case study considered i2S framework elements and how useful the research was perceived to be by next users. At the framework level (averaged across domains), all case studies gave moderate to strong consideration of i2S elements and were considered to have been at least moderately useful by next users. At a domain level, all the case studies gave at least moderate consideration to the elements of the i2S and were considered by next users to be at least moderately useful in all domains, with the exception of domain 2 in case study 5. Table 3. Consideration of i2S elements and usefulness of research process and outcomes by next users for seven case studies, scored on a 0-4 scale.    For all the case studies, there was greater consideration of i2S elements from domains 1 (synthesising science and stakeholder knowledge) and 3 (providing integrated research support) than from domain 2 (understanding and managing diverse unknowns). This was somewhat reflected in the usefulness assessments, as in five of the seven cases studies next users found research projects less useful for understanding and managing diverse unknowns than for synthesising knowledge or providing integrated research support.

Assessed Consideration of i2S Elements Assessed Usefulness by Next Users
Usefulness assessed by next users grew with increased consideration of i2S elements by the project teams. This relationship is seen both at the domain and whole-of-framework level ( Table 4). All the correlations are positive and significant (p < 0.05). The next-user interviews and surveys concluded by asking about the overall usefulness of the case study research processes and outputs. The responses from the overall usefulness section were analysed separately to cross-check against the results from the domain and framework evaluation ( Table 5). The correlation between the degree of consideration of the i2S elements and the overall assessment by next users was positive and significant (p < 0.05), supporting the domain and framework results. Table 4. Correlations between case study consideration of i2S elements and assessed usefulness of research process and out puts for seven case studies.

Variables Correlations
Consideration  Table 5. Correlation between case study consideration of i2S elements and assessed overall usefulness of research process and outputs for seven case studies.

Variables Correlations
Consideration of i2S elements: all framework Assessed usefulness by next users: overall assessment 0.76 (p = 0.045) There are two limitations in this study. Assessing the extent to which a case study considered the elements of the i2S framework may not explicitly capture the quality of the methods used. Although there is a significant correlation (0.74, data not shown here) between the extent of consideration of i2S elements and the project team's self-evaluation of how well they performed, assessing the extent of consideration only partially captures the quality of the methods used. The second limitation is that case study 4 only obtained next-user information from two next users, although the overall statistical findings were similar when case study 4 results were removed from the analysis.
These results suggest that the increasing consideration of the elements of the i2S framework produces more useful results for a range of next users and also suggest what to pay attention to when undertaking research on complex problems-the 'hand rails'. The i2S framework is not, however, prescriptive about the 'how'. The qualitative data, to which we now turn, provide some examples of what doing this type of research well looks like.

Results Part 2: The Aspects of the Case Studies That Next Users Found Most Useful
All of the case study projects did some things well. In this section, we use the next-user scores for the interview and survey questions to identify specific aspects of the case studies where the next users found the research activities or products most useful in each of the three i2S domains (scoring ≥ 3.5). We then describe what the relevant case study projects actually did.

Domain 1 Synthesising Disciplinary and Stakeholder Knowledge
The premise for the first domain in the i2S framework is the recognition that understanding and researching complex real-world problems does not just involve combining knowledge from multiple disciplines but includes relevant stakeholder knowledge as well. The aspects of the case studies the next users found most useful in domain 1 were how the projects had involved the right people and disciplines, how they had identified and understood the problem, and how they sought to understand multiple perspectives.

Involving the Right Disciplines and the Right People
The next users of case studies 2 and 3 considered that the projects had included all the necessary disciplines and stakeholders.
The projects used several methods to identify what knowledge sources and disciplines ought to be included. Case study 2, which was about on-farm nutrient losses, conducted some preliminary regional scale Geographical Information Systems (GIS) modelling of nutrient losses to empirically identify which industries (and therefore which disciplines) should be involved in the project. They noted that "those industries that represented a significant proportion (>80%) of the nitrogen loss in Canterbury were deemed to be most necessary to have involved". Case study 3, which was about farm nutrient management, considered the system they were trying to modify to identify the right disciplines to include. Through looking at the farm as a system, and the nitrogen cycle, they identified potential intervention points to target in the research and thereby identified the relevant disciplines.
To determine the right stakeholders to involve, case study 2 drew representations of stakeholder interest in and influence over the project to identify those individuals and organisations with the greatest influence and interest in achieving the project goal. Case study 3 undertook a similar stakeholder analysis, but in addition identified where and from whom farmers got their information, in order to include these key information sources. Both projects considered how change would occur as a result of the project and used this to identify others that needed to be engaged in the project.
In both cases, the project teams had a very clear purpose for bringing together different sources of knowledge and people: "we were trying to do science for policy and . . . the implementation was really key, so we wanted to have the people . . . most closely involved in implementation to be right in the heart of the research project making sure that it was credible and relevant and legitimate."(CS2) "These guys brought practical knowledge, combined with policy and research knowledge from the rest of the team."(CS2) "Needed to include various disciplines and end-users (farmers) to maintain integrity . . . and accelerate adoption."(CS3)

Identifying the Problem and Synthesising Knowledge
The next users of case study 7 found the research processes had been particularly effective for identifying the problem. This case study, which was about agricultural innovation, included researchers and agricultural industry partners and used an extensive range of systems tools to define the problem from multiple perspectives and investigate links between problem elements. As one project member recounted, "We used causal loop diagrams, CATWOE, innovation journey analysis, structures and functions framework, and multi-level perspective." (Causal loop diagrams are used in System Dynamics to understand the behaviour of a system over time. CATWOE is a method to prompt thinking about a change in the system. Innovation journey analysis is a method for analysing the history of an innovation from inception to implementation. Structures and functions framework is a design framework for designing a change in a system, comprising three classes of variables: function, behaviour, and structure. Multi-level perspective is a framework describing the scaling up and out of innovations from niche, to regime, to landscape scale.) For understanding the problem, the next users found the ways that case studies 2, 3, and 7 brought together and synthesised information from different knowledge sources particularly useful. All three projects used informal dialogue methods extensively. Through multiple workshops and discussions between researchers and stakeholders, participants gained a joint understanding of the system, of the context, and of the problem, and together interpreted the results.
In addition to the extensive use of dialogue, the three case studies used quite different approaches to bring knowledge together. Case study 2 co-developed a conceptual framework and then used a suite of numerical models to synthesise the different knowledge sources and populate the framework. Case study 3 employed place-based synthesis by using monitor farms to bring together the real-world experience of the farm and research experience and combined this with story-telling by different members of the project. Case study 7 synthesised the project findings for specific stakeholder needs, creating 'value-add' documents, where the research being done was brought together and interpreted with particular reference to each stakeholder perspective.

Understanding Different Perspectives
The next users of case studies 1, 2, and 6 reported that they had a good understanding of the diversity of perspectives and that the projects had managed those diverse perspectives well.
In case study 1, which supported a collaborative water policymaking process, the next users reflected that there had been a long and sometimes acrimonious history surrounding management of water, and it seemed as if each group had their own "facts, uncertainties, solutions and aspirations that never seemed to have been all tested together." To recognise the different perspectives but not privilege any one, the case study project team took a set of community values that had been developed and agreed by the community and used these values to guide technical assessment. They built an assessment framework that predicted the likelihood of a future scenario delivering a range of community values. Through the framework they could be "clear about reflecting the range of aspirations and comparing them completely uniformly".
For case study 2, there were six agricultural industries in the project who, together, were tasked with defining agricultural Good Management Practice (GMP). In this way, the project had already included a diversity of perspectives, across these different industries, when choosing whom to include. However, the project also recognised that there was important diversity within each of the industries. To capture this diversity in the development of GMP: "Each of the industries went away and did quite a significant canvassing and consultation and creation exercise with groups of farmers . . . to canvass perspectives within each of those groups. The project also included a pan sector reference group who were asked to look at things 'from a broad industry focus'." Case study 6, which was about improved irrigation management, used a visual aid to elicit and discuss a range of perspectives. They created a schematic diagram describing the irrigation management landscape. It conceptualised how people viewed irrigation in a wider context, explored who the different key players were, and what the barriers were at different scales. The project team "showed [the] map many times to get other perspectives: 'this is how I see it' and 'what's your view?' . . . when meeting with stakeholders, face-toface meetings, workshops at end of irrigation season". The schematic was updated as new perspectives were included.

Domain 2: Understanding and Managing Diverse Unknowns
The premise for the second i2S domain is that unknowns and uncertainties cannot be avoided. Complex real-world problems can be characterised as where " . . . facts are uncertain, values are in dispute, stakes are high and decisions are urgent" [64], and as the issues become more complex, the different dimensions of uncertainty become more apparent [46]. The next users of case studies 3, 4, and 7 found aspects of the way the projects tried to manage unknowns particularly useful. However, as there was less overall consideration of unknowns than of the other domains, the research activities pertaining to managing unknowns have been described together.
To be able to respond to unknowns as they arose, case study 3 built flexibility into the project. Instead of trying to decide what all of their research steps would be up front, they put 'go-no go' decision points into the contract to allow changes in direction throughout the project. They designed-in a mid-term stocktake of all their research to date. In this, they identified some promising options and this led to a refocussing of the research for the remainder of the project. They used this reflection process throughout the project by scheduling regular "research aim or critical step team meetings to reflect, analyse and (re)frame new research." Case study 4, which was about managing timber, used a similar adaptive project management approach to case study 3: "do reflect, adapt". As a way of getting a better understanding of the relevant unknowns and "to help mitigate for unknowns and uncertainty", the project used workshops to bring together multiple perspectives. In a series of industry engagement workshops, they used "debate and robust discussion" to reflect on the industry issues and the research being done. During one of these workshops, they made an important serendipitous finding, "only discovered by having an appropriate mix of research, development, grower and company in same room working together". The adaptive management approach they had adopted enabled them to adapt the programme in response to this discovery and go on to deliver benefits for the industry. Case study 7 began by completing a programme logic that "required considerations of assumptions and foreseeable unknowns that might arise". They kept track of emerging issues and unknowns from the beginning of the project using three approaches: (1) regular reflection sessions with the project team to explore why progress towards outcomes was working (or not), and they had dedicated reflexive monitors to help the teams through these reflection processes; (2) adaptive management to respond to the emergence of unknowns and maintain relevance; and (3) learning from successes and failures in other innovation projects.

Domain 3: Providing Integrated Research Support for Policy and Practice Change
The premise for the third domain is the increasing interest in how research influences policy and practice change. The aim is to do research that better supports decision-making, not only by dealing with what is known about a problem but also by helping better manage and account for unknowns and uncertainty. This domain also focuses the attention of researchers on how they can support the change they want to see happen as a result of the research. The aspects of the case studies the next users found most useful in domain 3 were relevant and timely outputs; well communicated outputs; and the way support had been given to the next users.

Producing Relevant and Timely Outputs
The next users of case study 6 thought the project team produced particularly timely and relevant results for them. In the project, farmers were clearly identified as the target audience and were critical to the project outcomes of improved irrigation management.
The project established what the farmers needed to help them manage their irrigation better. As a result, the project posted daily website updates and provided a portal for farmers to view their own farm data as well as data from other contributing farms. The project team also helped farmers make sense of their data through phone calls, training sessions, and emails.

Results Communicated Helpfully
The next users of case studies 2 and 3 found the research outputs were usefully communicated.
Case study 2 interacted with the target audiences from day one of the project and took an approach of extensive communication, using workshops, presentations, models, reports, briefs, one-on-one and group discussions, and attending policy development meetings to introduce the science where necessary: "Compared to a group of people going off doing a piece of research and delivering a report or a product [at the end] and walking away, it couldn't really have been further from that . . . having the next users on the team helps". They communicated the science at different levels of complexity. They made good use of different organisations, interests, and expertise in the team. They were mindful that team members "were credible or legitimate in different roles" and therefore involved "almost all of the members of the research team . . . in the delivery of the research findings in one way or the other, whether they were presenting to their peers, whether they were presenting to industry, or whether they were involved in the policy development." Case study 3 developed a clear communications plan for their target audiences and identified and planned communication events such as "workshops, field days and project team meetings". They also developed capacity within the host organisation to be able to integrate and communicate findings more widely.

Supporting Policy and Practice Change
The next users of case studies 1 and 3 found that the research processes and outputs were particularly useful for supporting them in policy or practice changes. In both case studies, the target audiences and the desired outcomes were clearly identified by the project teams, and right from the start both teams considered how they would support the implementation of the project results.
At the start of their project, the case study 1 team had "various discussions with [next users] about how we might best work together", including devices like field trips and "re-structuring the way we undertook the science in explicit recognition of the need to recognise end-users' needs and interests and values." The team were also very deliberate in how they framed their role: "We tried to frame the research findings as a 'service' trying to make transparent the consequences and uncertainties to allow [next users] to make an informed decision. We also resisted the invitations to 'tell us what the answer is'. The way we described [the results] was in the form of a coloured matrix with the likelihood of meeting the outcomes [desired] by the community. This meant that different scenarios could easily be compared." The team paid particular attention to who was providing support to the next users. Described as brokers, these individuals "were specifically identified as having skills, attributes and interest in translation and communication, [were] non-defensive in their approach and had empathy with target audience and were also comfortable with uncertainty".
In case study 3, before the project was funded, the team considered the changes they wanted to achieve and "workshopped the necessary steps to support farmers in transition" and used these insights to inform the project proposal. They used a programme logic approach to make transparent how they think change happens and tested it with the monitor farmers early on. To support these changes, the project endeavoured to have all "knowledge in the public domain [and] data available to all parties", which required them to resolve intellectual property issues between organisations in the early stages of the project. They also formed a relationship with the owners of an independent model that is used widely in the regulation of farm nutrient losses, to expedite the inclusion of the research in the model "in order to increase uptake and reward farmers for doing the right thing."

Discussion
The findings from this pilot suggest that the extent to which the concepts in the i2S framework are considered can influence the usefulness of research outputs to next users and, as such, provisionally suggest that the i2S concepts can be considered effective 'handrails through the swamp' in agri-environmental research (in particular, research projects that have multiple stakeholders, relevant scientific disciplines, and knowledge sources, and in cases where there is a real-world problem that is being tackled through policy or practice change, making management of unknowns a critical consideration). By examining the case studies through the lens of the i2S framework with insights from the next users, some useful examples of how these handrails are used to good effect surfaced.

Domain 1 Synthesising Disciplinary and Stakeholder Knowledge
The importance of acknowledging different sources of knowledge beyond traditional science disciplines is well recognized [5,30,34,65,66]. In domain 1, synthesising disciplinary and stakeholder knowledge, we see the strongest correlation between the case study research teams' use of concepts in the i2S framework and the perceived usefulness by next users. It is likely that the researchers involved already had experience in this area.
The case study projects considered most useful were very clear about why they needed different people, perspectives, expertise, and knowledge sources. The importance of this clarity is well recognised [67,68], and the i2S framework brings the question to the fore by asking, 'What is the research for and who is it intended to benefit?' What the framework is silent on, and what would be useful to include, is consideration at this early stage of who might lose out from the research [69].
The benefits of including other perspectives and knowledges envisaged by these case studies relate not only to gaining a better understanding of the system [34] but also practical legitimacy, increased rigour, and greater implementation [14,23,70]. Considering a breadth of perspectives contributes to the quality and impact of research according to Hansson and Polk [67], who also caution researchers to clearly identify which perspectives are needed and why, and not just go for the wholesale involvement of every player. Including both scoping and boundary setting in the i2s framework guides researchers to scope the disciplines, types of knowledge, and stakeholders that could be involved and then make transparent decisions on what and whom to include. Those case studies considered to have chosen stakeholders and disciplines well all used some type of independent analysis to help determine the disciplines and stakeholders needed.
How different types of knowledge are brought together is a principal challenge in integrated research [2,30,32]. The i2s framework acknowledges broad classes of synthesis such as dialogue and modelling. However, the case study projects deemed to have synthesised knowledge best used multiple methods of synthesis simultaneously. The case studies built on the benefits of individual synthesis methods such as modelling with stakeholder participation [17]; co-development [71]; place-based synthesis to anchor the project in the context of the user, e.g., [67]; and narratives to synthesise multiple perspectives [72] and communicate complex information [73], and combined the results in various ways to reflect that different people understand information in different ways, e.g., [74]. As no method of synthesis is complete, all focus on different aspects. Using multiple methods also has the advantage that they complement each other [75,76].

Domain 2 Understanding and Managing Diverse Unknowns
In domain 2, understanding and managing diverse unknowns, we see a weaker correlation between the case study research teams' use of concepts in the i2S framework and the perceived usefulness by next users (cf domain 1 and 3). Bammer [11] (p. 16) notes that this area of work generally receives much less attention in integrative research projects, and these case studies support this assertion, as all the case studies spent considerably less time and effort in activities of domain 2 than in domains 1 and 3.
Projects that were considered most successful in managing unknowns deliberately built adaptation into their project design. They recognised that impactful research must embrace uncertainty [8,47] and emergence [2]. They used reflection [4,77] and adaptive management [78] to respond to emergent situations or circumstances and serendipitous discoveries. Some projects used minimal specifications [8] in their contracting to support this flexibility and adaptiveness. An additional benefit of regular reflection was that it supported learning in the projects [7]. The i2S framework does not explicitly mention the importance of a learning approach, e.g., [8,23,38], but many of the projects deliberately created space for learning to occur between the research team, extended network, and next users, through devices such as field trips, gathering of a community of practice, or even something as simple as regular in-person gatherings of researchers, partners, and end users to jointly make sense of the research results.

Domain 3: Providing Integrated Research Support for Policy and Practice Change
In domain 3, integrated research support for policy and practice change, we see a strong correlation between the case study research teams' use of concepts in the i2S framework and the perceived usefulness by next users. Along with domain 1, this was a clear focus for the case study projects.
Those authors of the case studies that were considered most useful for supporting policy and practice change thought about the impact they wanted to have, the influence they wanted to exert, and how their project would support change, right from the start. They had a clear idea of the reasons why they needed to include a range of people, perspectives, and knowledge sources, and a clear idea of whom and what they were targeting and how those in that system made change. Instead of impact being considered at the end of a project and viewed largely as a communication exercise, these case studies designed the research around it, influencing what they did and whom they did it with. This designing for impact [7,30,79] has profound implications for the conception, design, implementation, and evaluation of individual research projects. Although the numbering of the domains might suggest that consideration of implementation and change happens towards the end of the project, Bammer (2013) notes that the framework is flexible in terms of the order in which it is used and it works best when implementation starts to be considered during project design.
The same case studies endeavoured to understand how their target audiences made decisions, including how they and other stakeholders considered unknowns. Lemos and Morehouse [71] note that useable science, as opposed to just useful science, directly reflects constituent needs, is understandable to users, is available at the time and place needed, and is expressed in a medium that is useful for next users. These case studies were attentive to the timing and frequency of decision-making, whether for policy or on the ground practice change. They explicitly sought to understand how different target audiences wanted to engage and what type of output would be meaningful for them. As Polk [21] found, the projects geared research synthesis and outputs to target audience needs to increase compatibility with the target audience's decision-making processes. The third domain in the i2S framework guides researchers to develop this greater understanding of the target audiences and their world.
In domain 3, researchers are prompted to consider not just the outputs of the research but also how that support is given. When choosing their teams, the case studies considered most useful viewed a range of skills and attributes to be important, such as brokering, integration, translation, reflection, communication skills, and the ability to work with difference [13,32,80], as well as credibility, empathy, humility, and an understanding of the decision-making and policy context [19].

Conclusions
For disciplinary scientists, the literature on integrative research remains fragmented and difficult to navigate. For this reason, we chose to evaluate an existing integrative research framework, the i2s framework, to try and provide disciplinary scientists with evidence of the framework's efficacy in tackling complex socio-environmental problems. We discovered that research teams who considered the concepts of the i2s framework produced research that was considered useful by their next users. These findings suggest that disciplinary scientists can use the i2s framework to generate useful agri-environmental research but that further evaluation of the i2s framework is necessary in applications beyond agri-environmental research to understand its broader efficacy for research in complex problems. Further research is also needed on when the framework is deployed in research projects to confirm its efficacy.

Data Availability Statement:
In accordance with the social ethics approval, the underlying data is not publicly available.

Conflicts of Interest:
The authors declare no competing interest.

Domain Question Element
Synthesising disciplinary and stakeholder knowledge What was the synthesis of disciplinary and stakeholder knowledge aiming to achieve and who is intended to benefit?
Purpose of combining disciplinary and Stakeholder (SH). Knowledge documented, Disciplinary and SH beneficiaries documented.
Which disciplinary and stakeholder knowledge was considered?
Systems tools used to help define the 'problem' and investigate links between problem elements. A recognised method used to develop a system view.
Identification of different views on defining the problem. Identification of all the disciplines that could be relevant. Identification of all the stakeholders that could be relevant. A recognised methodology used for identification of relevant stakeholders and disciplines. Relevant science disciplines included and clarity on how decisions were made on what research was undertaken.
Relevant stakeholders included and clarity on how decisions were made on which SH were considered relevant to include. All the relevant perspectives within each stakeholder group identified. Consideration given to how the problem was described or framed. Differences or conflicts managed.
How was the disciplinary and stakeholder knowledge synthesised, by whom, and when?
Knowledge from the stakeholder and science participants synthesised using either formal or informal methods. Consideration given to who was the most appropriate person (s) to lead the integration and synthesis. Synthesising of knowledge (stakeholder with stakeholder, discipline with discipline, and stakeholder with discipline) occurs at multiple stages in project.
What circumstances influenced the synthesis of disciplinary and stakeholder knowledge?
Overall context (and changes in context) influencing research project taken into account. Consideration given to the authorisation and legitimacy of the project. Project barriers and facilitators managed.
Understanding and managing diverse unknowns What was the understanding and management of diverse unknowns trying to achieve?
Clarity within research team, research stakeholders, and target audience on why managing for unknowns is important.
Systematic consideration of unknowns (at outset and on-going through project life). Opportunities and risks associated with undertaking the project managed.
How were the recognised unknowns and uncertainties managed or responded to?
Tools or processes used to help cope with uncertainty in research or implementation process, e.g., the precautionary principle, scenarios, sensitivity analysis, hedging, and adaptive management (acceptance approach). Identification and management of unknowns that were irreducible within the scope of the project. Surprises or the unexpected managed. Consideration of participants' tacit knowledge (researchers, research stakeholders, and target audience) and management of any conflicts. Communication of uncertainty in research outputs and implications for policy or practice change.

Domain Question Element
What circumstances influenced the management of unknowns Management of project barriers to and facilitators of understanding, and managing unknowns within research and target audience organisations.

III Providing integrated research support for policy and practice change
What is the integrated research support aiming to achieve, and who is intended to benefit?
Practice or policy change intent of the project clearly stated. Target audience (e.g., next-user, end-user) government, business, or civil society clearly stated.
Which aspects of policy and practice are targeted by the provision of integrated research support?
Consideration given to the target change as part of a system (e.g., if it was a change of regional council policy, was the way the regional council makes policy decisions considered, position in the policy cycle, reaction of lobby groups and community groups?) Identification of different systems views. Consideration given to how change happens-explicit or implicit (e.g., program logic, causal chain modelling, and theory of change). Identification of all the opportunities and target audiences relevant for the targeted change.
Consideration of how research findings were framed or described for the target audience.
Identified differences or conflicts in values between research team and target audience managed.
How was integrated research support provided, by whom, and when?
Consideration of what methods were used to deliver research findings to the target audience. Consideration given to who was involved in the delivery of the research findings.
Interaction with the target audience occurs at multiple stages in project.
What circumstances might/did influence the provision of integrated research support for policy and practice change?
Overall context (and changes in context) influencing target change and target audience managed. Consideration given to the authorisation and legitimacy of the project to interact with target audience. Identification and management of difficulties in moving from doing the research (domain 1) in to trying to implement (domain 3). Project barriers and facilitators with target audience managed.

= moderately familiar 4 = very familiar
My frequency of engagement with the XXX research team was: 0 = never 1 = rarely 2 = occasionally/sometimes 3 = a moderate amount 4 = a great deal Domain1: Synthesising stakeholder and disciplinary knowledge As a next or end-user of the XXX project please rate how useful the knowledge synthesis and research outputs were for understanding the problem and proposed solutions. Not at all useful = 0, moderately useful = 2, very useful = 4 As a next or end-user of the XXX project, please rate how useful the knowledge synthesis and research outputs were for helping you understanding the perspectives and values of all the affected or interested parties. Not at all useful = 0, moderately useful = 2, and very useful = 4 Was the project relevant to Māori? NA/Yes. If yes, how useful was the integration of mātauranga Māori into the research outputs for understanding the Māori perspective to the research problem and proposed solutions? Not at all useful = 0, moderately useful = 2, very useful = 4 How usefully was the perspective of your industry sector (next-, end-user stakeholder group) reflected in the research knowledge synthesis and outputs? Not at all useful = 0, moderately useful = 2, very useful = 4 Were any affected or interested parties and/or disciplinary knowledge sources not well represented or excluded from the research outputs? If yes, who and why should they have been better represented?
Domain 2: Understanding and managing diverse unknowns How useful was the analysis of uncertainties and unknowns, which might influence the problem issue and proposed solutions, in the research outputs? Not at all useful = 0, moderately useful = 2, and very useful = 4 How useful was the communication of uncertainties and unknowns, in the research outputs, for use in policy development or practice implementation? Not at all useful = 0, moderately useful = 2, very useful = 4 Were any uncertainties or unknowns that you considered important not addressed or inadequately addressed in the research outputs? What were they and how do you think this impacted the usefulness of the research outputs?
Domain 3: support for policy/practice implementation How useful were the research project outputs for supporting policy development or practice implementation? Not at all useful = 0, moderately useful = 2, and very useful = 4 How useful were the research outputs in term of providing timely, relevant information for policy-makers or practitioners? Not at all useful = 0, moderately useful = 2, and very useful = 4 How useful were the research outputs communicated for policy development or practice implementation? Not at all useful = 0, moderately useful = 2, and very useful = 4 If mātauranga Māori was relevant to the project, how useful were the research outputs for integrating mātauranga Māori into policy development or practice change? Not at all useful = 0, moderately useful = 2, and very useful = 4, NA How objectively were the research results presented? Not at all objective = 0, moderately objective = 2, and very objective = 4 Additional review questions What is your overall evaluation of the usefulness of the research project outputs for policy development or practice implementation? Not at all useful = 0, moderately useful = 2, and very useful = 4 How legitimate was the research process and outputs? Not at all legitimate = 0, moderately legitimate = 2, and very legitimate = 4 How credible were the research results and outputs? Not at all credible = 0, moderately credible = 2, and very credible = 4 How relevant were the research results to subsequent policy development or practice implementation? Not at all relevant = 0, moderately relevant = 2, and very relevant = 4 What would have led to a greater sense of legitimacy, credibility, and relevance of the research outputs?


Answer Generation for Retrieval-based Question Answering Systems

Recent advancements in transformer-based models have greatly improved the ability of Question Answering (QA) systems to provide correct answers; in particular, answer sentence selection (AS2) models, core components of retrieval-based systems, have achieved impressive results. While generally effective, these models fail to provide a satisfying answer when all retrieved candidates are of poor quality, even if they contain correct information. In AS2, models are trained to select the best answer sentence among a set of candidates retrieved for a given question. In this work, we propose to generate answers from a set of AS2 top candidates. Rather than selecting the best candidate, we train a sequence to sequence transformer model to generate an answer from a candidate set. Our tests on three English AS2 datasets show improvement up to 32 absolute points in accuracy over the state of the art.

Introduction
Question answering systems are a core component of many commercial applications, ranging from task-based dialog systems to general purpose virtual assistants, e.g., Google Home, Amazon Alexa, and Siri. Among the many approaches for QA, AS2 has attracted significant attention in the last few years (Tymoshenko and Moschitti, 2018;Tian et al., 2020;Garg et al., 2020;Han et al., 2021). Under this framework, for a given question, a retrieval system is first used to obtain and rank a set of supporting passages; then, an AS2 model is used to estimate the likelihood of each sentence extracted from passages to be a correct answer, returning the one with the highest probability. This approach is favored in virtual assistant systems because full sentences are more likely to include the * This work was completed while the author was an intern at Amazon Alexa.
Q: How a water pump works? c1: A small, electrically powered pump. c2: A large, electrically driven pump (electropump) for waterworks near the Hengsteysee, Germany. c3: A pump is a device that moves fluids (liquids or gases), or sometimes slurries, by mechanical action. c4: Pumps can be classified into three major groups according to the method they use to move the fluid: direct lift, displacement, and gravity pumps. c5: Pumps operate by some mechanism (typically reciprocating or rotary), and consume energy to perform mechanical work by moving the fluid.
G: A water pump is a device that moves fluids by mechanical action. Table 1: An example of a question Q and five answer candidates c 1 , . . . , c 5 from WikiQA (Yang et al., 2015) ranked by an AS2 system. Answer G generated by our best system is significantly more natural and concise than any extracted candidates.
right context and sound natural, both of which are characteristics users value (Berdasco et al., 2019). AS2 models have shown great performance on academic benchmarks. However, these datasets fail to consider many essential qualities of a QA system which interacts directly with users, such as a virtual assistant. In some cases, extracted answer sentences contain the correct information, but the focus of the answer doesn't match the question; in others, the answer requires reasoning or contextual knowledge from the user or is very long and contains extraneous information. For example, in WikiQA (Yang et al., 2015), a widely used AS2 dataset, the answer "Wind power is the conversion of wind energy into a useful form of energy, such as using wind turbines to make electrical power, windmills for mechanical power, wind pumps for water pumping... " is considered a good answer for "What can be powered by wind?", even though its formulation is burdensome to a user.
In this work, we explore a fundamentally dif-ferent approach to AS2. Rather than selecting the best candidate, we propose using a model to generate a suitable response for a user question. In so doing, we extend the traditional AS2 pipeline with a final generation stage that can recover correct and satisfying answers in cases where a ranking AS2 model fails to place an acceptable candidate at the top position or where a top candidate with the desired information is not a natural-sounding response to the query. Table 1 shows an example of our system: given the question, Q, and a list of candidates, C k = {c 1 , . . . , c 5 } sorted by a state-of-the-art AS2 system, we use a sequence-tosequence model to produce an answer G given Q and C k as input. This approach, which we refer to as GenQA, addresses the limitations of AS2 systems by composing concise answers which may contain information from multiple sources.
Recent works have shown that large, transformer-based conditional generative models can be used to significantly improve parsing Rongali et al., 2020), retrieval (De Cao et al., 2020;Pradeep et al., 2021), and classification tasks (Raffel et al., 2019). Our approach builds on top of this line of work by designing and testing generative models for AS2-based QA systems. In recent years, the use of generative approaches has been evaluated for other QA tasks, such as machine reading (MR) (Izacard and Grave, 2021;Lewis et al., 2020b) and question-based summarization (QS) (Iida et al., 2019;Goodwin et al., 2020;Deng et al., 2020).
However, while related, these efforts are fundamentally different from the experimental setting described in this paper. Given a question, generative MR models are used to extract a short span (1-5 tokens) from a passage that could be used to construct an answer to a question. In contrast, AS2 returns a complete sentence that could be directly returned to a user.
QS systems are designed to create a general summary given a question and one or more related documents. Unlike QS, AS2-based QA systems need to provide specific answers; thus, the presence of even a small amount of unrelated information in a response could cause the answer sentence to be unsuitable. In contrast, we show that our approach can succinctly generate the correct information from a set of highly relevant sentence candidates.
In summary, our contribution is four-fold: (i) we introduce a new approach for AS2-based QA systems, which generates, rather than selects, an answer sentence; (ii) we illustrate how to adapt state-of-the-art models such as T5 (Raffel et al., 2019) and BART (Lewis et al., 2020a) for answer generation; (iii) we show 1 that our GenQA system improves over the state-of-the-art AS2-based systems by up to 32 accuracy points, as evaluated by human annotators; finally, (iv) we briefly explain why traditional generation metrics are not suited for evaluating AS2-based systems.

Datasets
We use four English datasets in our work, one related to generative QA and three to AS2. For a fair comparison between selector and generation methods, we re-evaluate the top answers returned by all models using a fixed set of annotators. All annotations were completed by company associates who are not part of our research group and had no knowledge of the systems. Annotators were required to mark an answer as correct if it was: (i) factually correct; (ii) natural-sounding; and (iii) required no additional information to be understood. All QA pairs were single annotated, as we determined sufficient agreement for this task in previous campaigns.
WikiQA by Yang et al. (2015) contains queries from Bing search logs and candidate answer sentences extracted from a relevant Wikipedia page. For evaluation, we used the dev. and test sets, which contain 126 and 243 unique questions and we re-annotated all of the resulting 569 QA pairs. 2 Answer Sentence Natural Questions (ASNQ) introduced by Garg et al. (2020) was derived from the NQ dataset (Kwiatkowski et al., 2019) and consists of the questions which have a short answer span within a single sentence in a long answer span. The sentences containing the short answer are marked as correct and the other sentences in the document are marked as incorrect. We use the dev. and test splits introduced by Soldaini and Moschitti (2020) which contain 1,336 questions each. We re-annotated a total of 5,344 QA pairs.
WQA is an internal AS2 dataset created from a non-representative sample of questions asked by users of a virtual personal assistant in 2019 3 . For each question, we retrieved 500 pages from an index containing over 100M web documents. We then ranked candidate answers using a state-of-theart AS2 system, and annotated up to 100 of them. In total, the training and dev. sets contain 3,074 queries and 189k QA pairs, while the test set contains 808 queries. For this effort, we re-annotated 4,847 QA pairs from the test set.
MS MARCO QA NLG (MSNLG) by Nguyen et al. (2016) is a subset of the MS MARCO dataset focused on generating natural language answers to user queries from web search result passages. It consists of 182k queries from Bing search logs, the ten most relevant passages retrieved for each query, and a well-formed answer synthesized by an annotator. This dataset is not designed for AS2, but it represents a large resource of succinct and clear answers, thus making it close to our AS2 task.

Generative QA Model (GenQA)
The AS2 task is defined as follows: Let q be an element of the question set, Q, and C q = {c 1 , . . . , c n } be a set of candidates for q, e.g., sentences retrieved by a search engine, where c i ∈ C, and C is a set of candidates. We model a selector S : Q × C n → C, such that S(q, C q ) = argmax i (p(q, c i )), where p(q, c i ) is the probability that c i is a good answer. We also define S k : Q × C n → C k , such that, S k selects the top k answers in descending order of p(q, c i ).
State of the Art Throughout our experiments, we use TANDA (Garg et al., 2020) as our state-ofthe-art selector S. This AS2 model was trained as a binary classifier on (q, c i ) pairs using a sequential fine-tuning approach starting with ASNQ and finishing on a target dataset, e.g., WikiQA. Specifically, we use their pretrained RoBERTa Large model , as it achieved the best results on all datasets it was tested on.

Our Generative Approach
Instead of selecting the best candidate, we generate a new answer using the information from the top k answer candidates. Thus, our model is a function G : Q×C k → G, where G is the text that can be generated by the generator G from the question, any fragment of the retrieval set, the model's vocabulary, and knowledge stored in the model's parameters. Formally: The example in Table 1 shows that we can generate a correct answer from a set of candidates which, as a whole, contain enough information to formulate a correct answer. We propose that a valid answer can be built by composing the most promising constituents coming from the different candidates in C k . Intuitively, information repeated across multiple candidates is more promising; therefore, we hypothesize that a model trained on the same or similar generation task should be able to effectively exploit this form of repetition, even in cases where the same information is presented in a similar, but not identical manner. Further, recent works have shown that large transformer models hold a substantial amount of commonsense knowledge in their parameters (Roberts et al., 2020), which our model could leverage to perform inference across sentences in C k , e.g., associate water with fluid in the example in Table 1.

Fine-tuning GenQA
Given a pre-trained transformer seq2seq model, e.g., T5 (Raffel et al., 2019) or BART (Lewis et al., 2020a), we obtain G by fine-tuning on a large AS2 or QA generation dataset. For this purpose, we format our training data as a standard sequence-to-sequence/text-to-text task, where the source text is the question concatenated with the top five answer candidates, (q, S k=5 ), joined by newlines. When an answer composed by a human is available, such as in MSNLG, we use it as the output target. For cases where there is no composed answer, we randomly select a known-good candidate to be the target, remove it from the inputs and replace it with another candidate if one is available. We truncate the input text to 512 tokens and, at test time, we use beam search with a beam size of four and a maximum output length of 100 tokens.

Experiments
In this section, we first report on our experimental setup, then we show the results on fine-tuning GenQA, and finally, we report on the comparative results between AS2 and GenQA.

Setup
Models and Parameterization Our GenQA model is based on the T5 (Raffel et al., 2019) variant of the UnifiedQA (UQAT5) model by Khashabi et al. (2020). We use the Large version of UQAT5, which has 770M parameters for all of our experiments. We compute training loss as the mean of the cross-entropy between the softmax probabilities over the output vocabulary and the one-hot encoded target answer. We finetune UQAT5 with a learning rate of 5E −5 . We also experiment with the Large variant of BART (Lewis et al., 2020a), which is comprised of 400M parameters. This model was trained using same loss with learning rate 5E −6 .
Evaluation We used accuracy as our primary metric for all our experiments and models. This is computed as the average number of questions a model answers correctly; for a selector S, it is equivalent to Precision at 1. For S, we also report Hit Rate at 5, which is the fraction of queries with at least one good candidate ranked five or less.
Beside human evaluation, we also experimented with automatic evaluation metrics such as BLEU (Papineni et al., 2002) and ROUGE-L (Lin, 2004) for GenQA. Such metrics have found little success in evaluating QA tasks (Chaganty et al., 2018;, so we investigate whether that is the case for AS2 as well.

Results
How to Fine-tune GenQA? As described in Section 4.1, we tested two GenQA variants: one uses a UnifiedQA T5 (UQAT5) (Khashabi et al., 2020) as base model, while the other leverages BART-Large (Lewis et al., 2020a). Of the datasets used in this work, MSNLG and WQA are large enough for fine-tuning GenQA. Therefore, based on preliminary results, we tested four different strategies for training UQAT5: fine tuning on (i) WQA or (ii) MSNLG alone, (iii) combine the two datasets by alternating mini-batches during training, or (iv) follow the transfer-then-adapt strategy proposed by Garg et al. (2020): first fine-tune on MSNLG, then adapt to a AS2 using WQA. Table 2 reports the results on the WQA test set, which are all relative to the performance of the state-of-the-art model (TANDA). First, we ob-  serve that all GenQA models reported in this table considerably outperform the best selector model, TANDA. This result shows that our generative approach can improve system based on AS2.
Comparing the accuracy of different training strategies applied to UQAT5, we achieve the best results when the model is trained on MSNLG alone (+19.9% over TANDA baseline). While we were initially surprised by this result, as MSNLG is not designed for AS2, error analysis suggests that GenQA benefits from the high quality training data (concise answers written by annotators). Conversely, when training with WQA, we observed that GenQA tends to produce answers that, while correct, are not as natural-sounding. We plan to explore how to best leverage existing AS2 datasets for generative model training in future work. We also note that a GenQA BART-Large achieves comparable results to GenQA UQAT5 on WQA; in preliminary experiments, we found training strategies reported on UQAT5 to have similar effect on BART-Large.
When manually annotating results of our early tests, we found that BART was more likely to be extractive and copy input passages in their entirety while UQAT5 was more likely to compose new text and produce answers with textual overlap from multiple input candidates but was more likely to hallucinate content. We found that through hyperparameter tuning we could largely eliminate the hallucination from UQAT5 answers but we were unable to make BART more abstractive.
Similar to what has been observed in other QA tasks (Chaganty et al., 2018;, we find that automatic metrics do not correlate with assessments from human annotators. This is ASNQTEST 69.0 87.9 37.9 ± 51.5 90.5 13.9 ± 5.6 Table 3: Accuracy of our GenQA UQAT5 model compared to a state-of-the-art AS2 model by Garg et al. (2020). All answer candidates returned by the two models were re-annotated to ensure a fair comparison. Length is the average number of tokens in the answer.
due to the fact that neither BLEU nor ROUGE-L are designed to estimate whether an answer is clear and natural-sounding, instead rewarding candidates that have high overlap with reference answers. Most importantly, such overlap is a poor indicator of factual correctness.
Comparison between AS2 and GenQA Table 3 reports the results of TANDA and GenQA on two standard AS2 datasets, evaluated with manual annotation. We note that there is an impressive gap of over 20 absolute accuracy points on both development and test sets. This result is produced by two important properties of GenQA. First, it builds correct answers from a pool of correct and incorrect answers, and it can generate a good answer so long as the relevant information can be found anywhere in the top k = 5 candidates. This is a clear advantage over using TANDA alone, as Hit-Rate@5 of 99.2%, and 87.9% for WikiQA and ASNQ, respectively, ensures that GenQA often receives at least one correct answer as input. Second, GenQA exhibits the ability to rewrite unnatural answers from a text snippet into an answer suitable for a conversation. For example, for the question "What year did Isaac Newton die?", TANDA returns candidate "Sir Isaac Newton (25 December 1642-20 March 1727 was an English physicist and mathematician". Although correct, no human would provide it in such a form. In contrast, GenQA composes a concise answer: "Isaac Newton died in 1727".
Finally, Table 3 shows that the size of GenQA answers, in terms of words, is only 14 tokens, which is 2.7 times less than the 30-40 tokens from TANDA. This further suggests that GenQA can provide more concise and direct answers, which are preferable in a conversational context.

Conclusions
In this work we present GenQA, a generative approach for AS2-based QA systems. The main difference with recent MR-based generative systems is the capacity of our models to generate long answers. This comes from the use of AS2 candidates (complete sentences) as input to our generative approach. In contrast, MR systems, being mainly trained with short answers, e.g., noun phrases and named entities, mostly generate short answers.
We show that GenQA significantly outperforms state-of-the-art selector models for AS2 by up to 32 accuracy points by combining different pieces of information from the top k answer candidates. These results suggest promising directions for generative retrieval-based systems.