Value of agreement in decision analysis: Concept, measures and application

In multi-criteria decision analysis workshops, participants often appraise the options individually before discussing the scoring as a group. The individual appraisals lead to score ranges within which the group then seeks the necessary agreement to identify their preferred option. Preference programming enables some options to be identified as dominated even before the group agrees on a precise scoring for them. Workshop participants usually face time pressure to make a decision. Decision support can be provided by flagging options for which further agreement on their scores seems particularly valuable. By valuable, we mean the opportunity to identify other options as dominated (using preference programming) without having their precise scores agreed beforehand. The present paper quantifies this Value of Agreement and extends the concept to portfolio decision analysis and criterion weights. The new concept is validated through a case study in recruitment.


Introduction
Multi-criteria decision analysis (MCDA) often assesses options with an additive independent value function (Keeney & Raiffa 1976). The performance of each option ∈ is valued on each criterion ∈ along a 0-100 scale to obtain the value scores (see Table 1 for symbols). While the value score for tangible criteria can often be derived from well-defined, marginal value functions, which map attribute levels to value scores (e.g. frequency of a service, horsepower of a machine, traffic noise), many intangible criteria require highly subjective judgements to assign value scores to options (e.g. impact of constructions on scenery, level of expertise of organisation, comfort of vehicle) (Keeney 1982). The criteria are weighted against each other, leading to the relative importance judgements with ∑ = 1. The decision model recommends the option with the greatest score = ∑ .
Real-world problems are rarely that simplistic, and extensions to the model are therefore required. The present paper addresses MCDA problems in which the score preferences are initially incomplete; i.e. the criterion scores can take any number in the range [ , ̅ ]. The options' value scores hence span from = ∑ to ̅ = ∑ ̅ . If there is one option that dominates all others, it is called the robust option (Roy 1998).
Such incomplete preference information (Hazen 1986, Kirkwood & Sarin 1985, Sarin 1977 is often encountered by decision-making groups appraising options against intangible selection criteria. In multicriteria group decision making, participants frequently appraise the options individually before seeking agreement on the scores. The individual precise criterion scores by participants ∈ lead to criterion score ranges from = min{ } to ̅ = max{ }, within which the group looks for the necessary agreement to identify the group's robust option. 'Aggregation' and 'consensus' are the two principle methods to reach this necessary agreement. Using the aggregation method, an influence weight is assigned to each participant (e.g. Dias & Sarabando 2012, Kim & Ahn 1999, Salo 1995. A precise group score for all decision options can be automatically calculated by weighting the individual scores, which immediately clarifies the group's preferred option according to the participants' influence. Using the consensus method, the group engages in a discussion from which the necessary consensus on the scores needs to emerge with mutual agreement (e.g. Dias & Climaco 2005, Matsatsinis & Samaras 2001, Phillips 2011, Shakhsi-Niaei et al. 2011, Vilkkumaa et al. 2014. The aggregation method stresses the positional power of each decision maker, while the consensus method emphasises mutual learning from each other's insights in the decision problem. When using the consensus method, time pressure may make it challenging for the group to come to a full agreement on all scores (Kim & Ahn 1997, Weber 1987. In this paper, we develop a new concept, which helps the consensus method in multi-criteria group decision-making to be more time-effective.
When thoroughly reviewing the individual appraisals = ∑ and the resulting score ranges from = min{ } to ̅ = max{ }, a skilled facilitator might be able to roughly guess which options are definitely dominated, which options are likely to be dominated after further agreement on some other options is sought, which ones have a totally unclear fate and which ones have a good chance of becoming the robust one. The present paper attempts to replace this intuitive guessing with a quantitative concept we call Value of Agreement. Options with a high Value of Agreement are more attractive to a group discussion as knowing their agreed value score may allow eliminating other options as dominated without the need to seek agreement on their scores beforehand; thus, finding the robust option will require less effort. Altogether, the Value of Agreement estimates the impact of eliciting additional preference information on the time requirements of a decision analysis workshop.
Designing workshops to eliminate options quickly has hitherto been addressed only by very few publications. Hämäläinen & Pöyhönen (1996) and Salo & Hämäläinen (1995) analysed problems where only the weight information was incomplete. They suggested that the group should first seek agreement on the criteria with large weight ranges.  and  provided decision aid to wisely choose the next swap in the MCDA method even swaps.  studied which criterion should be used as the reference criterion for the weight elicitation to harness preference programming effectively. Liesiö et al. (2007Liesiö et al. ( , 2008 briefly examined the problem for portfolio decision analysis but only offered limited guidance for an option elicitation order. 1 The paper proceeds as follows: Section 2 briefly discusses normative preference programming techniques to identify definitely dominated options. Section 3 explains descriptive approaches developed by psychologists to predict on which precise scores * the group may finally agree. Section 4 pulls together the normative and descriptive decision-making perspectives from the previous two sections and develops a prescriptive measure for the Value of Agreement on option scores in MCDA. Section 5 modifies this measure for multi-criteria portfolio decision analysis (PDA)-an important extension of MCDA. Section 6 adapts the Value of Agreement to weights. Section 7 demonstrates the effectiveness of the proposed measures with a computer simulation. Section 8 reports on the application of the Value of Agreement concept to a PDA-based recruitment selection problem using computer software. Section 9 concludes with a research outlook.

Preference programming
Preference programming encompasses a set of techniques to eliminate definitely dominated options and ultimately identify the robust option given the incomplete information about scores and weights (Arbel 1989, Punkka & Salo 2013, Salo & Hämäläinen 1992. The expressed incomplete preferences are typically range-based or ordinal-based (e.g. Kirkwood & Sarin 1985, Liesiö et al. 2007, Punkka & Salo 2013. Preference programming is a normative approach that, in its original form, captures the incomplete information in a set of linear constraints. Non-dominated options can be easily identified by examining the extreme points of the resulting convex hull (e.g. Hazen 1986, Liesiö et al.  White et al. 1984) or by solving mathematical programmes (e.g. Arbel & Vargas 1993, Lahdelma et al. 1998, Salo & Hämäläinen 1995. The range-based strict dominance rule (e.g. Hazen 1986) 1 ≻ 2 ⇔ 1 ≥ ̅ 2 and ̅ 1 > ̅ 2 is a logic based on preference ranges which can be applied in a basic MCDA problem where just the score information is incomplete. For the robust option * , the strict preference * ≻ must hold for all ∈ \{ * }. 2 An ordinal-based weak dominance rule can be constructed by considering participants' implicit rankings when assigning scores to options. Option 1 weakly dominates option 2 on criterion if all participants believe that 1 ≥ 2 . In this case, the linear constraint 1 ≥ 2 can be added to the preference programme. Option 1 weakly dominates options 2 overall if all participants believe that 1 = ∑ 1 ≥ ∑ 2 = 2 . In this case, the linear constraint ∑ 1 ≥ ∑ 2 can be added to the preference programme. Moreover, assume there is already a group agreement that 1 ≥ 2 . In this condition, learning that the group agreed on the precise number 1 ( 2 ) for the score 1 ( 2 ) may allow for tightening the upper (lower) bound such that ̅ 2 ≔ min{ 1 , ̅ 2 } ( 1 ≔ max{ 1 , 2 }). The same tightening rule can be used for option scores. If there is a group consensus on 1 ≥ 2 and the participants agree on the precise option score 1 ( 2 ), the upper (lower) bound for 2 ( 1 ) can be updated to ̅ 2 ≔ min{ 1 , ̅ 2 } ( 1 ≔ max{ 1 , 2 }).

Predicting agreement
The group's actual agreement on a precise number for a score can be predicted when all group members have already made their individual appraisals . Next, we distinguish between the case in which the participants only know the single appraisals or their ranges [ , ̅ ] (section 3.1) and the case in which an aggregate of their appraisals such as the mean or the median is presented to the participants (section 3.2).

Social judgement scheme
The social decision scheme (Davis 1973) has led to much insight into how small groups arrive from individual preference statements to a group decision along a discrete scale of measurement (Parks & Kerr 1999). A major finding of this research stream is that individuals whose preferences are quite similar to those of other group members wield an exponentially greater influence on the final decision than those individuals with extreme preferences. Davis (1996) manifests this observation in the equations of his social judgement scheme (SJS)-a continuous generalisation of the discrete social decision scheme.
denotes the SJS centrality position of group member for the assessment of score and is defined as can be interpreted as the percentage of influence of assessor on the group's collective choice for . * = ∑ is the most central value for according to SJS, and thus a good point estimator of the group's unknown actual agreement . Fig. 1 exhibits an example where the point estimator * is derived from individual appraisals using three common aggregates and SJS.
Wide research on the social decision scheme provides a solid theoretical foundation for the centrality position approach. Furthermore, experiments by Bonner et al. (2004), Davis et al. (1997), Nadler et al. (2001) and Ohtsubo et al. (2002) offer good evidence in favour of SJS. Applications of SJS in decision support systems for MCDA workshops can be found in the studies conducted by Rigopoulos (2008), Tsiporkova & Boeva (2006) and Tundjungsari et al. (2012). It is important to also highlight that SJS shares the property of heavily discounting extreme opinions with the geometric mean as the consensus estimator. While there are many case studies that use the geometric mean in group decision problems modelled as Analytical Hierarchy Process (e.g. Saaty & Vargas 2001, Vaidya & Kumar 2006, we are not aware of any research that validates the geometric mean as an appropriate consensus estimator. It needs to be mentioned that the strength of SJS is in majority/plurality situations where no group member can plausibly demonstrate why their judgement must be the correct one. Given that preference statements are rather subjective (e.g. Keeney 1992, Payne et al. 1993, Slovic & Lichtenstein 2006, this assumption is normally fulfilled for the MCDA problem. In situations where members with extreme opinions can (i) plausibly demonstrate the correctness of their positions, (ii) have hierarchal power or (iii) are recognised as the group's experts, the centrality concept of SJS is not the right approach (Bonner et al. 2007, see also : Laughlin 1980, Laughlin & Ellis 1986.

Proposed agreement
In some consensus-based MCDA workshops, the group is presented with a proposed aggregate of the individual appraisals to support and expedite its quest for a consensus. For instance, the mean/median is often seen as a good estimator of the unknown 'optimal' value (called 'wisdom of crowds' by Surowiecki 2005, see also: Ariely et al. 2000, Sunstein 2005, from which the group should only diverge with good reasons. It is also possible to have more sophisticated proposed aggregates that consider fairness concerns among group members or uneven influence of participants on the collective decision based on positional power or expertise (e.g. Keeney 2013, van den Honert 2001. Viewing the proposed aggregate prompts an anchoring heuristic (Tversky & Kahneman 1974) and it becomes likely that the group finally agrees on this presented value. For example, we observed in a recent decision analysis workshop at the National Audit Office in Britain that the participants chose the median shown to them beforehand for 31 out of 35 options. Hence, the predicted agreement * should take the value of the proposed aggregate in such situations.

Measures for the Value of Agreement in MCDA
Our proposed measure for the Value of Agreement rests on the assumption that one can make reasonably precise predictions * = ∑ * on what scores the group will agree on for options , as outlined in the previous section. The score ranges [ , ̅ ] and the predicted agreements * can be graphically presented in a value score chart. It will be shown that such a chart helps define two types of measures, denoted and henceforth, indicating the likelihood of eliminating dominated options without reaching prior full agreement on them. Bringing the two measures together in a lexicographical fashion leads to a recommendation on which options the group should next seek agreement.
In a value score chart (see Fig. 2), the options are displayed in the form of horizontal lines along a scale of 0-100. The horizontal lines represent the option score ranges [ , ̅ ] and the dots on them show the predicted agreements * . The borderline between the red area (appears in darker grey on black-andwhite prints) and the green area (appears in light grey) marks the potentially still unknown agreed precise value score for the still unknown robust option. ̂= max{ * } is a heuristic choice for . * is always replaced by = ∑ if the group has already elicited full agreement on option . Note that the heuristic choice ̂ underestimates systematically, as the expected value of the maximum of a set of random variables is larger than the maximum of the expected values of the same set of random variables (Chung 1948). There is a research opportunity to explore distribution functions for the agreement rather than just settling for a point estimator * . Such work would be interesting in itself and could also lead to unbiased heuristic choices for .
For those options , whose bars are partially in the value score chart's green area, i.e. ∈ ( , ̅ ), it is not at all certain whether one can identify them as dominated without further agreement on their value scores. To quickly eliminate dominated options, options whose hypothetical agreements * would not only shift themselves, but also other options outside the green area (or on its edge 3 ) due to ordinalbased weak dominance relations would therefore be particularly attractive to a group discussion. Let [ ′ # , ̅ ′ # ] be the overall score range of option ′ ∈ after having learnt that option takes the predicted value score * , then the count is a simple measure for the Value of Agreement for option . takes the value of the heuristic choice ̂ in our computer simulation in section 7.3. Assume, for instance, that the participants collectively agree   that option 4 is better than option 2 on a particular criterion in Fig. 2. Knowing the actually agreed criterion score 4 for option 4 may allow pushing the upper bound of option 2 leftwards when using the tightening rule from section 2, such that ̅ 2 #4 ≤ . This, in turn would make it much more likely to rule out that option 2 requires further agreement for it to be identified as dominated. In this example, (4) = 1. 4 In many cases, the agreement on ordinal preference relations is not sufficient to allow shifting a further option besides off the green area after learning that = * . A tie-breaker thus is needed: if the upper bound of option is to the far right from the borderline , i.e.
( ) = ̅ − is large, it is less likely that the group can avoid some further agreement on its score. Thus, it would be wise to discuss those options with large first. For the example in Fig. 2, recommends choosing option 1 next.
Pulling the two measures and together leads to the lexicographic rule argmax for selecting the next option for group discussion. In words: choose the option that maximises ( ) and if more than one possible option exists, then select among those the option that maximises ( ).

Measures for the Value of Agreement in PDA
Portfolio decision analysis (PDA) extends MCDA by allowing the selection of more than one option (Phillips & Bana e Costa 2007, Salo et al. 2011). In the basic form of the PDA problem, fixed costs are attached to each option (typically called 'project' in PDA), options can be chosen independently of each other, and the total number of options is limited by a budget . A robust portfolio * has at least an as high an overall score of ∑ ∈ * (and for one precise score allocation, a higher overall score) as that of any other feasible portfolio independently of which scores the group agrees on within the remaining ranges [ , ̅ ]. The PDA is a special knapsack optimisation problem (Martello & Toth 1990), in which the majority of cases favour options with a high value-for-money ratio = / (Phillips & Bana e Costa 2007). The preference programme from Liesiö et al. (2007) is currently the only available knapsack optimisation approach for incomplete score information that allows the identification of options which are definitely or definitely not in the robust portfolio. The remainder of this section will derive a measure for the Value of Agreement analogously to the MCDA case.
In a value-for-money chart (see Fig. 3), the options are displayed in the form of horizontal ratio bars representing the feasible ratios ∈ [ / , ̅ / ]. When ordering the options by their value-for-money ratios, one can calculate the borderline value-for-money ratio , defined as the mean of the value-formoney ratio of the last option that still fits completely and the first option that fits only partially in the portfolio (see Fig. 4). A heuristic choice ̂ for the borderline ratio can be computed by using the predicted value-for money-ratios * = * / . is the border between the red and green areas in the valuefor-money chart.
For options crossing the borderline ratio, i.e. ∈ [ , ̅ ], it is quite unclear whether they will be in the robust portfolio. Again, options whose predicted agreement * would not only shift themselves, but also other options away from the borderline due to ordinal preference relations should be considered first for the group discussion. Let [ ′ # , ̅ ′ # ] be the updated overall score range of option ′ ∈ assuming that takes the predicted option score * , then provides a first measure for the Value of Agreement for option . Assume, for example, that the collective agreement is that options 5 and 6 are both better than option 8 in Fig. 3. In such a case, knowing the agreement on option 8 makes it likely that options 5 and 6 can be ruled out as requiring further agreement to identify them as definitely in the robust portfolio. In this example, 1 (8) = 6.
As a tie-breaker, the distance between the borderline ratio and the nearest end of the option's ratio bar, i.e.
, is used. If 's ratio bar does not intersect the borderline ratio, ( ) obtains a negative value. The larger the of an option, the less likely it is that the group can avoid some further agreement on the option's score. In Fig. 3, the measure recommends option 1 next.
The next option to be discussed by the group is again chosen using the lexicographic rule argmax It should be noted that the two measures for the Value of Agreement described in this paper work only for the basic PDA problem. Model extensions, such as incomplete cost information, options that cannot be independently chosen or an additional constraint on the maximum number of options in the portfolio would require different measures for the Value of Agreement.

Extension to weights
In this section, we adapt the previously developed measures for the Value of Agreement to SWING weighting with incomplete weight information. The value score information is assumed to be complete in the following calculations and assessors only appraise the weights for all criteria individually. We also assume that the popular SWING weighting method (Edwards & Barron 1994, von Winterfeldt & Edwards 1986) is employed and that the decision making group jointly agrees on a reference criterion as a first step. A fixed, non-normalised weight (e.g. 100 points) is assigned to reference criterion what establishes a linear scale of measurement. Each participant assigns non-normalised individual weights ̃ to all other criteria ∈ \{ }, reflecting their importance relative to the reference criterion. The resulting non-normalised weight ranges span from ̃= min{̃} to ̃= max{̃} within which the group seeks actual agreements ̃. Since the non-normalised individual weights ̃ and the actual agreements ̃ are all based on the same scale of measurement, established by the reference criterion , the Value of Agreement for SWING weights can be computed in a similar method as for scores.
The weight ranges [̃,̃] form a convex hull of feasible weight combinations. To compute the minimum and maximum overall value score of options, only as the set of extreme points of this convex hull is relevant (Hazen 1986, Liesiö et al. 2007). Without loss of generality, it is assumed in the following that the weight combinations {w j } j∈J in are always normalised such that they sum up to 1; i.e. ∑ = 1 ∀{ } ∈ ∈ . An option's overall value score lies between In the following, * denotes the predicted weight agreement for criterion derived analogously to section 3 and takes the value after a group agreement has been reached. The predicted weights * are normalised and sum up to 1.
Since this paper only aims to introduce the new concept Value of Agreement, applications to other weight elicitation methods are not explored further. For instance, applying the Value of Agreement to the AHP (Saaty 1990) would require prioritising the elicitation of weight ratio pairs and addressing the issue of inconsistencies. For the point assignment method (Nutt 1980), the fact that the actual agreedupon weights must add up to a constant sum (e.g. 100 points) and therefore cannot be elicited independently from each other poses a challenge to the feasibility of applying the Value of Agreement concept. Finally, if the decision makers choose to structure a larger number of criteria in a value tree (Keeney & Raiffa 1976), any Value of Agreement measure would have to take into account the interdependencies of the weights on different hierarchy levels.

Preference programming
As a range-based strict dominance rule, we use the linear programming approach from Hannan (1981) for the MCDA case and the dynamic programming approach from Liesiö et al. (2007) 5 for the PDA case. The ordinal-based weak dominance rule for weights is analogous to value scores (see section 2).

MCDA
The heuristic choice for the borderline is max{∑ * }. Let [ # , ̅ # ] be the overall score range of option ∈ after having learnt that criterion takes the predicted weight * and having applied the ordinal-based weak dominance rule. The first measure for the Value of Agreement, counts the number of options whose upper bounds are still larger than the heuristic choice after assuming = * . In general, favours 'high impact' criteria with a large normalised weight span − and large variance Var { } among its criterion scores. Using ( ) = max{ ̅ # − } as a tie-breaker, the lexicographic rule for selecting the next criterion to be discussed by the group is argmax ∈ ( ) ≫ argmax ∈ ( ) .

PDA
The borderline value-for-money ratio is computed as described in section 5 using * = ∑ * as the predicted agreement. [ # , ̅ # ] is the updated overall score range after assuming = * . The first measure for the Value of Agreement is again favouring 'high impact' criteria. The normalised weight range is used as a tie-breaker; i.e. is used to choose the next criterion to be discussed by the group.

Test dataset
Several authors (e.g. Keisler 2004, Salo & Hämäläinen 2001, Vetschera et al. 2014) have previously deployed extensive computer simulation as a means to test ideas on preference programming and value of information. To demonstrate the effectiveness of the proposed measures for the Value of Agreement, a large test dataset with 360 instances was generated (see Table 2 and Supplementary Material A). The dataset comprises 20 instances for each possible combination of number of options, score distribution, weight distribution and cost distribution as provided in Table 2.
The group members' individual appraisals are constructed as follows:  Two participants 1 ∈ and 2 ∈ are always randomly chosen and are assigned the upper and lower criterion score bound; i.e. 1 = and 2 = ̅ .  is normally distributed with ( +̅ 2 , ( The group members' actual agreements are modelled by two different distributions A1 and A2. The more predictable median-focused distribution A1 is triangular with ( , * , ̅ ). If the originally predicted agreement * is smaller than or larger than ̅ due to bound tightening, it takes the value of the bound or ̅ , respectively. The less predictable distribution A2 is uniform with ( , ̅ ). For each combination of properties from Table 2, 10 instances receive actual agreement scores according to A1 and the other 10 according to A2. Starting with 11 , the actual agreement scores are computed one after the other. After every assignment of a new actual agreement score , the feasible ranges [ ′ ̅ ′ ] for all still non-précised scores are tightened as described in section 2. After full actual agreement about a further option is constructed, the ranges [ ′ , ̅ ′ ] of all still non-précised options are tightened as described in section 2. If is outside the tightened score ranges for option , the whole procedure is restarted from 11 . The actual weight agreement is modelled analogously.

Comparison measures
To evaluate the effectiveness of the proposed measure for the Value of Agreement, we compare it with the simple rules of MaxRange and Random. The MaxRange rule (cf. Hämäläinen & Pöyhönen 1996, Salo & Hämäläinen 1995) enquires the precise actual agreement of the option (criterion) with the greatest current score range [ , ̅ ] (weight range [ , ]) until the robust option or portfolio is elicited. The Random rule retrieves the precise actual agreement for a randomly chosen option (criterion) for which ( ) has not yet been revealed. Preference programming 6 is applied for both rules after every single actual agreement to exclude options that do not require further preference elicitation.
By enumerating all feasible elicitation orders, it is possible to compute a lower bound for the number of required elicitations for each test instance. This lower bound could always be reached if one knew the participants' actual agreements in advance. As the number of criteria is small in the test dataset (| | = 5), a full enumeration can be performed for all instances with incomplete weight information. Designing a branch-and-bound enumeration algorithm to compute the lower bound efficiently for all instances with incomplete score information might be possible, but beyond the scope of this paper.

Results
The test algorithm always selects the option or criterion with the greatest Value of Agreement, reveals its modelled actual group agreement, and applies preference programming 6 on the new situation. The median is used as predicted agreement for scores and weights. Tables 3 to 6 show the results for the test dataset.
In the MCDA case, the Value of Agreement rule on average reduces the number of option (criterion) elicitations statistically significantly by 50% (8%) and 45% (19%) compared to the MaxRange and the Random rules, respectively. For incomplete score information, MaxRange consistently ignores options with small ranges on the right-hand side of the value score chart despite the fact that these are more likely to be the robust option. Therefore, the Random rule outperforms the MaxRange rule. In the PDA case, the Value of Agreement rule on average decreases the number of inspected options (criteria) statistically significantly by 41% (2%) and 47% (12%) relative to the MaxRange and the Random rules, respectively. V0: ~(0,100) and = ̅ ∀ , (complete score information) V1: ~(0,95) and + 5 = ̅ in 25% of the cases ~(0,90) and + 10 = ̅ in 25% of the cases ~(0,85) and + 15 = ̅ in 25% of the cases ~(0,80) and + 20 = ̅ in 25% of the cases V2: ~(0,90) and + 10 = ̅ in 25% of the cases ~(0,80) and + 20 = ̅ in 25% of the cases ~(0,70) and + 30 = ̅ in 25% of the cases ~(0,60) and + 40 = ̅ in 25% of the cases The resulting ′ from settings C1 and C2 are normalised to so that ∑ = 2,500. = 1,000 is assumed as the budget for all problem instances.
The results underscore that a more accurate prediction of the agreed value score improves the effectiveness of the Value of Agreement rule. For A2 instances, the Value of Agreement rule decreases the number of required option agreements by 41% in the MCDA and by 43% in the PDA case, compared to the Random rule; but for A1 instances, these numbers rise to 49% in the MCDA and 51% in the PDA case. Overall, the Value of Agreement measure seems to make the greatest gains by looking for options that trigger ordinal-based weak dominance rules, have a large score range and are not too far away from the true borderline . The gains from a more precise prediction of the agreement (i.e. changing from A1 to A2 instances) were much smaller. This is because prediction errors are limited by the scores' upper and lower bounds, what means that the predicted borderline ̂ should even for less accurate predictions of the agreed scores not be located too far from the unknown true borderline in most cases.
Increasing the score ranges from setting V1 to V2 leads to 66% and 33% more required option agreements for the MCDA and PDA case, respectively, when using the Value of Agreement rule. This is an expected result because the more the participants disagree on the scoring, the longer it should take to reach group agreement on the robust option. Decreasing the spread of the options' value-for-money ratios from setting C2 to C1 doubles the average number of required option agreements in the PDA case when using the Value of Agreement rule. This result is also less surprising because if all options tend to be, on average, more or less equally attractive in terms of their value-for-money ratios, it becomes less likely that many options are dominated or dominating, independent of what actual scores the group finally agrees on.
Unlike for incomplete score information, the reduction in required preference elicitations is small for incomplete weight information-both in percentage and nominal terms. This can be explained by the fact that the MaxRange rule already provides solutions near the lower bound. The required number of elicitations when using the MaxRange rule is just 29% and 8% higher for the MCDA and the PDA case, re-    spectively, as compared to the lower bound. The Value of Agreement rule reduces this deviation from the lower bound to 19% and 6%, respectively.

Application to recruitment
Ennovators Ltd. is a medium size e-commerce brand developer with approximately 100 employees in its London office. In summer 2015, the company's marketing department advertised to recruit a pricing analyst. The recruitment exercise was conducted by a panel consisting of the head of e-commerce, a marketing manager and a pricing analyst. A decision analysis model informed the recruitment process.
Several real-world applications of decision analysis to the recruitment process have been previously reported in the literature. Most publications showcase a fuzzy approach (e.g. Afshari et al. 2013, Petrovic-Lazarevic 2001, Polychroniou & Giannikos 2009); often combined with other techniques such as AHP (Celik et al. 2009, Khosla et al. 2009), ANP (Kabak et al. 2012) and TOPSIS (Kelemenis & Askounis 2010, Wang et al. 2006. Using fuzzy sets, the authors seek to consider the uncertainties associated with appraising candidates. Stal-Le Cardinal et al. (2011) applied Electre Tri to select a portfolio of students for university admission. Gibney & Shang (2007) used AHP for recruiting the dean of a business school. Finally, Gardiner & Armstrong-Wright (2000) provided detailed insights into developing a simple MDCA model for faculty recruitment in view of anti-discrimination laws. In line with Barclay (2001), we could not find evidence in the literature showing that decision analysis is used much in practice to inform the recruitment process. Among other reasons (Jessop 2004), we argue that the complexity of the elicitation process in many previous case studies severely hinders the uptake by practitioners (cf. Belton & Stewart 2002, Keeney & von Winterfeldt 2007. Therefore, it was our aim to design a user-friendly, time-efficient and rigorous decision analysis approach for the Ennovators' recruitment process. As the initial step of the Ennovators' recruitment process, the panel jointly constructed the decision model by agreeing on criteria, value scales and weights. Based on the job description and the company's core behavioural values, 'analytics software', 'numerical skills', 'personality fit', 'pricing experience' and 'self-motivation' were chosen as the assessment criteria. For each criterion, a candidate's minimum required performance level in order to be considered for the job was assigned a value score of 0 points and 100 points for the maximum desired performance level. Verbal descriptions of the performance levels were assigned to the value score points of 25, 50 and 75 for each criterion using the bi-section method (Fishburn 1967). The resulting value scales are shown in Table 8. Note that the verbal descriptions of the five equal-distanced points on the value scales only serve as a reference, and panel members were free to assess candidates with any other value score between 0 and 100 for a particular criterion. All five criteria are intangible in the sense that a candidate's precise value score for a criterion depends on the assessors' subjective interpretations of CV, cover letter and interview performance based on the assessors' knowledge and experience. Due to the diversity of job candidates, it is not practical to construct more precise value scales (Gardiner & Armstrong-Wright 2000). Criteria trade-offs were assessed using the SWING weighting method (Edwards & Barron 1994, von Winterfeldt & Edwards 1986 which compares the value of improvement from 0 to 100 for each criterion. The recruitment process takes the form of a portfolio decision analysis problem with a predetermined number of candidates being invited to the next interview round (the budget) and each interview causing fixed costs of 1.
An easy-to-use software was developed to support preference programming under incomplete score information for portfolio decision analysis (freely available from Supplementary Material B). Fig. 5 shows a screenshot of the software's window for the value-for-money chart from the case study. Candidates who will not and will definitely be invited for an interview are displayed on the bottom left and right, respectively. The Value of Agreement concept suggests discussing Marco next, whose value score bar 7 is highlighted. Double-clicking on the name opens a new window in which the decision makers can see the score ranges [ ̅ ] and they can type in their jointly-agreed scores for Marco. The software tool allows the choice between the social judgement scheme (used in this case study) and the VOGA's candidate review consisted of four steps. In step 1, HR screened the 87 incoming applications and excluded 47 (e.g. no proper cover letter, requiring work permit, no university degree, obvious Table 7 Assessment criteria with weights in parenthesis and value scale.

Analytics software (21%)
100_ Advanced skills in Excel (e.g. pivot tables, conditional formatting) AND considerable experience in VBA AND experience in Google Analytics AND considerable experience in at least one other relevant analytics tool (e.g. data mining software, R, SQL, computer programming) 75_ Advanced skills in Excel AND experience in VBA AND experience in at least one other relevant analytics tools 50_ Advanced skills in Excel AND some experience in VBA 25_ Advanced skills in Excel 0_ Basic skills in Excel (e.g. chart building, cell functions)

Numerical skills (24%)
100_ Degree in a highly numerical subject (e.g. math, science, finance, economics) AND at least one year's work experience in an analyst role 75_ Degree in a highly numerical subject OR at least one year's work experience in an analyst role 50_ Degree in business administration, marketing or a related field OR some experience in applying numerical skills in a work environment 25_ Degree which includes some numerical modules (e.g. psychology) 0_ Non-numerical university degree Personality fit 8 (15%): Creative, good command over English, experience working at a SME, open-minded, outgoing, pro-active, sense of humour 100_ Candidate is expected to fit perfectly into the culture of Ennovators AND most of the staff are very much expected to like to work with the candidate (evidence to look for: previous work experience of dealing with people, examples of behaviour provided in cover letter and interview, and verbal and non-verbal communication in the interview) 75_ Candidate is expected to fit well into the culture of Ennovators with some minor limitations (e.g. minor language barriers, no evidence of dealing successful with conflicts, a bit shy) 50_ Candidate is not expected to contribute positively or negatively to Ennovators' working environment AND candidate is unlikely to cause any communication or behavioural problems 25_ Other employees are expected to be able to work together with the candidate without any major issues AND candidate has room for improving communication and social interactions (e.g. very shy, less efficient communication) 0_ Staff at Ennovators could work with the candidate (e.g. rather poor English, rather unskilled communicator, slightly arrogant)

Pricing experience (29%)
100_ Experienced in developing pricing models or strategies 75_ Familiar with pricing through work experience 50_ Good understanding of pricing theory from university 25_ Some basic understanding of pricing (e.g. through marketing modules at university) 0_ No previous knowledge median to predict the agreed value score. To avoid biasing the decision makers, the software's valuefor-money chart does not reveal the predicted scores * to the participants.
Ennovators' candidate review consisted of four steps. In step one, HR screened the 87 incoming applications and excluded 47 (e.g. no proper cover letter, requiring work permit, no university degree, obvious misfits). In step two, each panel member scored the remaining 40 applications individually using the agreed value scales in Table 7. In a subsequent one-hour group discussion, the panel members sought the necessary agreement on the scores to select the 10 candidates who should be invited to an interview (individual and agreed value scores are available in Supplementary Material C). The discussion was facilitated by a decision analyst who also entered the agreed scores into the software tool. At no point during the discussion did the panel members have a particular preference on which candidate's score agreement should be sought next, and therefore they always followed the suggestion made by the software tool. In step three, nine interviews were held soon afterwards (one candidate sent their apologies) and each panel member individually updated their previous scorings. In the subsequent group discussion, the panel members used the software tool to agree on the three candidates to be invited to the final round on the following day; this discussion was however conducted without the support from a decision analyst. In step four, a problem-solving interview was held and the panel members updated their previous individual scorings of the candidates before discussing them as a group. The software tool was not employed in step four because the panel required a full ranking of the three candidates in case the best one rejected the job offer. Overall, the panel members were confident that they made the job offer to the best candidate, which meant that the decision model was requisite (Phillips 1984).
The case study revealed two technical problems that were not relevant in the computer simulation. The first problem involves candidates on whom the panel members disagree with regards to their individual scorings; i.e. whether or not a particular candidate meets the minimum required performance level for a particular criterion. If a panel member believes that a candidate does not meet the minimum expected performance levels in criterion , they would choose a prohibitive high value score of = −300 such that candidate can practically never become a 'dominating option' unless all the panel members jointly agree on a better score for following a group discussion. The second problem entails situations where the group agrees on a value score for option in criterion which is outside its range [ , ̅ ]. This situation occurred once in the case study when none of the panel members realised before the group discussion that one candidate's personality fit was actually very poor. Some previously determined dominance relations may become invalid if the agreed value score of lies outside the range [ ̅ ]-thus, options previously identified as dominated or dominating need to be reconsidered. In cases where an option 's agreed value score on a particular criterion lies outside its original value score range, takes the value for all participants and the preference programme is reinitialised, beginning with the complete original set of options.
Reflecting on the recruitment process, the panel members highlighted that the use of decision analysis forced them to properly assess candidates against all job requirements and to try to avoid becoming biased by individual aspects (see also Gardiner & Armstrong-Wright 2000, Gibney & Shang 2007. Although considerably more time-consuming, the panel members agreed that using proper multi-criteria decision models should lead to better and more transparent recruitment decisions. Since scoring models (in various forms and shapes) are commonly used in a business environment, the panel members readily grasped and accepted the core elements of the developed decision model (additive value function, bisection method, and SWING weighting).
The panel members also believed that the often-employed aggregation method to reach consensus in multi-criteria group decision making would have been inappropriate in this case study. As mentioned in the introduction section of this paper, the aggregation method computes a proposed aggregate based on group members' individual scores and automatically decides on the group's preferred portfolio using the scores . But several times, only one panel member ′ made a particular observation about a candidate (e.g. noticing an important but hidden detail in the CV or knowing a particular analytics software that the candidate mentioned) which led to a rather extreme criterion score ′ ; however, this turned out to be justified in the subsequent group discussion in the sense that the actually agreed score was close to ′ and far from the predicted score * . Thus, calculating the preferred candidate portfolio based on automatically calculated aggregates could have led to a poor recruitment decision in this case study. On the other hand, it would not have been practically feasible to discuss all 40 applications one-by-one due to time constraints. Therefore, preference programming offered a way to retain valuable individual scores by not settling for automatic aggregates but also keeping the decision problem at a practically manageable size. In this context, the Value of Agreement can be seen as a useful tool to make preference programming more effective but also more user-friendly in situations where a skilled facilitator is not present.

Conclusion
Deciding on which option a group should discuss next in a decision analysis workshop following individual option appraisals has always involved only informed guess-work. This paper introduced the Value of Agreement concept to measure for which options it is particularly worth seeking further group agreement in order to make a robust group decision quickly. An extensive computer simulation and a case study demonstrated the usefulness of the proposed Value of Agreement concept. In addition, we provided a free software programme to allow practitioners and researchers to readily employ preference programming and the Value of Agreement for standard portfolio decision analysis.
The Value of Agreement concept is primarily developed for decision problems where the number of options is large, but the available time to seek group agreement is rather limited. Practical examples include recurring decision problems such as hiring processes, maintenance prioritisations, project tenders, credit approvals and research funding decisions. As most decision problems are assessed with only a few criteria, we are less convinced of the practical relevance of applying the Value of Agreement to weights. Furthermore, by comparing the results for three different prioritisation rules (Value of Agreement, MaxRange and Random) with the theoretical lower bound, the possibilities for reducing the number of required elicitations were shown to be limited for incomplete weight information.
We broadly defined the Value of Agreement as an estimate for the impact of eliciting additional preference information on the time requirements of a decision analysis workshop. The present paper applies the new concept only to a specific workshop design and problem formulation. Therefore, there is scope for more research. First, the present paper assumes that the group always seeks maximal agreement on an option's value score before discussing the next option. Future work may advocate a more flexible approach of switching between discussing the individual criterion value scores from different options. In this case, the Value of Agreement concept could be used to provide guidance on which criterion value scores seeking further agreement is particularly worthwhile and when it might be advisable to move on to the next option without having reached full agreement on the option's value score . Thereby facilitators need to be mindful of the additional cognitive burden to participants that might be imposed by the frequent switching of the discussed option. Second, the present paper assumes that a group is prepared to discard an option only if it is strictly dominated. However, in some cases, the maximum loss of value in discarding a particular option may be so small that it would not be worth the decision making group's time (Salo & Hämäläinen 2010). Thus, future applications of the Value of Agreement concept might explicitly consider the costs of decision makers' time when proposing the next option for group discussion. Ultimately, the Value of Agreement concept might be used to seek group consensus on preference assessments until the maximum loss of value of the decision problem has been reduced to a level that the group feels comfortable with. The final decision could be made later by using the aggregation method mentioned in the introduction section, by applying decision rules for incomplete preferences such as minimax regret or by delegating the final decision to particular individuals for further considerations. Third, the present paper assumes that decision makers express their preferences numerically. Alternatively, the Value of Agreement concept can also be applied to qualitative preference judgements (e.g. Fasolo & Bana e Costa 2014). Those qualitative judgements can be in the form of preference classes (e.g. 'very good'), ordinal preference relations (e.g. 'X belongs to the 5 best options') or intensity preferences (e.g. 'weakly preferred'). A behind-the-scenes model translates the qualitative judgements into incomplete preference information (e.g. Bana e Costa & Vansnick 1994, Punkka & Salo 2013) and solves the resulting preference programme. Again, the question is where the decision maker should begin to make their qualitative judgements more precise. Fourth, the present paper applies the Value of Agreement concept only to SWING weighting. If future research can demonstrate that the Value of Agreement is indeed practically useful for weights, the adaption of this concept to other weight-elicitation methods would be an obvious research opportunity. Fifth, the present paper only addresses a basic PDA formulation. Applications of the Value of Agreement concept to more general PDA cases would be welcomed.
Future work may also develop measures for the Value of Agreement in a more systematic way than in this paper. Better heuristic choices for the predicted score of the robust option in the MCDA case and better usage of the knapsack problem characteristic in the PDA case would be desirable. It is also possible to examine whether there are benefits from recommending the next option for group discussion based on more complex option agreement scenarios. For instance, a decision tree could be used to model how the choice for the next option is influenced by which options the group might select afterwards and what happens if the group agreement deviates from the prediction. Finally, a more axiomatic approach to the constructions of measures for the Value of Agreement, with proven properties of the solution quality, would be of theoretical interest.
As a final point, it should not be concluded from this paper that the group must always mechanically attempt to find full agreement on the option with the greatest Value of Agreement. Sometimes there are options for which agreement can be found very easily or those on which the participants have fundamentally different perspectives . Dias (2007) argues that the participants should first focus on the less difficult elicitation questions to allow them to learn more about the used method and their preferences before tackling the difficult ones. One may also want to consider that gaining almost no progress in finding agreement from the start can endanger the 'emotional life' (Phillips & Phillips 1993) of the group and thus, the success of the workshop. Therefore a skilled facilitator should balance the recommendations prescribed by the Value of Agreement and the need to tackle easy tasks first.

Acknowledgement:
The author is grateful to Dr Gilberto Montibeller for suggesting the research topic and Trang Le for making the case study at Ennovators Ltd. possible. He also thanks Prof Ahti Salo and Dr Andrew Wilshere for their helpful feedback.

1_
Following their argument, options with a core index of around 0.5 should probably be examined first. 2 As an alternative to Hazen's strict dominance rule, one may apply a quasi-dominance rule instead (Dias & Climaco 2005). In this case, an option * is defined as robust if no other option ∈ \{ * } with an upper bound ̅ exceeding the lower bound * of option * by more than a given tolerance value exists. Quasi-dominance allows taking into account that the maximum loss of value the group could suffer when choosing option * may be too small to justify letting the decision making group convene for longer (Salo & Hämäläinen 2010). Quasi-dominance would add more complexity to the decision model and therefore is not discussed further in this paper.

3_
When the heuristic choice ̂ takes the value of an already agreed overall value score of an option (i.e. ̂= ), the upper bounds of some other options might be tightened such that ̅ = =̂. Those options are highly unlikely to be identified as robust after further preference elicitation. Therefore, the open interval ( ′ # , ̅ ′ # ) is used for ( ) in the MCDA case so that options are counted as practically dominated.

4_
The table below provides a numerical illustration for the example depicted in Fig. 2. Three participants assess four options on two criteria and the median is used to predict the group's agreement. The borderline takes the value ̂= max{ * } = 66. Furthermore, as an additional notation, let ̅ ′ # be the upper bound of ′ when assuming that takes the predicted value * .
Criterion 1 ( 1 = 0.5) Criterion 2 ( 2 = 0. It is shown in the following that the measure for the Value Agreement of option 4 takes the value of 1. Option 4 dominates option 2 on criterion 2 because all three participants assigned a higher score to 42 than to 22 . Using the median, the predicted score for option 4 on criterion 2 is 42 * = 38, which becomes the upper bound for option 2 on criterion 2; i.e. 22 #4 = 38. 21 #4 = 21 = 90 because option 4 does not dominate option 2 on criterion 1. It follows that the upper bound for option 2, when assuming the predicted agreement for option 4, is ̅ 2 #4 = 1 21 #4 + 2 22 #4 = 64, which is smaller than the borderline ̂. The upper bound of option 3 ̅ 3 is already smaller than ̂, and ̅ 4 #4 = 4 * = 58 as well. There are no dominance relationships between option 1 and 4, which means ̅ 1 #4 = ̅ 1 = 97. It follows that only option 1 intersects the borderline when assuming that the group agrees on the predicted scores * for option 4, meaning that (4) = 1.

5_
When testing Liesiö et al.'s PDA preference programme (2007) for incomplete weight information, we noticed that their algorithm does not identify all dominated portfolios and therewith all dominated options. For instance, it is possible that a portfolio 1 is dominated by portfolio 2 or 3 for all feasible weight combinations, but there is always a feasible weight combination such that 1 is not dominated by 2 and a feasible weight combination such that 1 is not dominated by 3 . As Liesiö et al.'s algorithm only uses pairwise comparisons of portfolios, it does not recognise that 1 is actually dominated. To test whether a portfolio 1 is dominated by any other set of portfolios for all feasible weight combinations, one can solve a linear programme with arbitrary objective function where (i) the weights are constrained by their upper and lower bounds and (ii) there is a linear constraint for each other's portfolio such that 1 is not dominated by this portfolio. If the linear programme has no feasible solution for the weights, there is no feasible weight allocation so that 1 is not dominated-hence 1 is dominated. 6_ For the PDA case, the preference programme from Liesiö et al. (2007) is given a time limit of 2 seconds. The algorithm is written in VB.NET x64 release and ran on a 2.8 GHz processor with 4GB RAM. 7_ As = 1 for all candidates ∈ , the value-for-money ratio bars depicted in Fig. 5 are identical with the candidates' value score bars [ , ̅ ].

8_
Given the limited information about a candidate's personality fit before the interview, the panel members agreed to be generous when appraising only the candidates' written applications and scored most participants between 70 and 90 on this criterion in step 2 of the recruitment process.