Loss aversion as a motivator for engagement with peer assessment

ABSTRACT Loss aversion has been shown to influence decision making in a host of social and economic contexts. This paper presents a novel assessment strategy that seeks to address issues of student engagement and academic integrity through the combination of the concepts of loss aversion and peer-assessment. Students faced the possibility of losing a small number of the marks they had previously ‘earned’ in a task in which they were required to accurately grade a peer’s work. A detailed rubric was designed to support students grading the work. Based on data collected via a survey and interviews, we show that students, despite disliking this assessment strategy, were more engaged in the assessment and felt that their critical thinking skills were enhanced as a result of this approach. The implications from this study includes the need for detailed guidance for students through, for example, rubrics, lecturer accessibility and exemplars.


Introduction
There are competing assessment purposes which can sometimes cause tension. Boud (2000, p. 160) encourages to consider assessment in terms of having a 'double duty', or more than one purpose. He urges us to judge assessment ' . . . in terms of its effect on learners and learning ' (2000, p. 155). Newton (2007) calls on educators to be clear about the purpose of assessment and ensure that the system prioritise the purpose. The assessment strategy presented in this paper was designed based on Boud and Associates (2010, p. 1) proposition that '[T]he reason for an explicit focus on improving assessment practice is the huge impact it has on the quality of learning'; and to reflect our universities assessment policy where academics are advised that '[A]ssessment and feedback can frame and influence a student's approach to learning and learning goals and his/ her motivation to engage in and with learning activities' (DCU, 2013).
The aim of the experiment in assessment practice outlined in this paper was to improve student engagement in a medium-sized (approx. 100 students) final year undergraduate economics module. This module is 100% Continuous Assessment (CA) and was typically assessed through two essay questions. A number of students each year were failing to engage with this task, and 2-3 full weeks were being consumed correcting approx. 240 essays. Of more relevance, the lecturer felt that the students were not as engaged as they could be in the class and approached members of Dublin City University (DCU) Teaching Enhancement Unit (TEU). Together they devised an innovative assessment strategy that blended the concept of loss aversion (discussed below) with the mechanism of peer assessment. Technology was used to leverage the approach in a more efficient and effective manner, specifically the Virtual Learning Environment (VLE) rubric and workshop activities.
Another influencing factor in designing this assessment strategy was feedback from students saying they were over-assessed. Elkington (2016) asks academics to consider whether we are over assessing learners, and in this context to design a more integrated and holistic approach to teaching, learning and assessment. In Ireland, the National Forum for the Enhancement of Teaching and Learning in Higher Education (National Forum) undertook research across thirty undergraduate degree programmes and the findings ' . . . indicate that the average number of assessments for single-semester modules was almost equal to that for full-year modules, which strongly suggests that modularisation tends to give rise to an increased number of assessments' (National Forum, 2017, p. 32). A typical stage or year for a learner is two fifteen week semesters, so a full year module is thirty weeks. As this is a five ECTS credit final year undergraduate module, it fits into this profile. One of the impetus for the re-designed assessment strategy was to ensure the assessment fit for purpose and authentic.
In the next section, we review the theoretical underpinnings of peer assessment and loss aversion. We then outline the assessment process and technology used to facilitate an efficient deployment of the strategy before outlining our mixed-methods approach and discussing our results. The final section concludes and offers some suggestions for practice and further research.

Loss aversion and peer assessment
The assessment process, outlined in this paper, was informed by the concept of loss aversion. Loss aversion is a powerful source of bias in decision making. Kahneman and Tversky (1979) in their seminal contribution, note that 'losses loom larger than gains' (p. 279) when people are considering changes in their welfare. In other words, people put a larger weight on potential losses than they do on potential gains when deciding upon a course of action. Kahneman and Tversky demonstrate this by showing that people on average need the promised winnings from a toss of a fair coin to be at least twice the losses if they guess incorrectly. Pope and Schweitzer (2011) show that this effect of reference points is also evident in high stakes situations in their study of a professional golfer putting behaviour. Players trying to make par (the reference point for not losing a stroke) display more effort than those trying to get under par. Hossain and List (2012) document that the framing of a bonus scheme can lead to better outcomes. The same bonus incentive framed as a punishment for not meeting a target as opposed to a reward for meeting targets led to an extra 1% gain in productivity in a Chinese high-tech manufacturing plant. The potential for loss aversion to be leveraged to facilitate better outcomes is clear.
The theory of loss aversion has also been found to have explanatory power and policy relevance in education. Wang and Yang (2003) examine how loss aversion can see competitive incentives for students lead to low effort provision as they adopt a 'strategy to win by not losing' (p. 126). Fryer, Levitt, List, and Sadoff (2012) examine teacher effort and show that mathematics test scores are higher for students with teachers who are paid upfront and must repay the money if their students do not improve. Levitt, List, Neckermann, and Sadoff (2016) also find some evidence that students respond more to rewards framed as losses. Of greatest relevance to the design of our assessment, Apostolova-Mihaylova, Cooper, Hoyt, and Marshall (2015) find that framing the grading system in terms of losses leads to better outcomes, though only for males. Women in the study performed worse under the loss aversion frame. McEvoy (2016) shows that statistics students who have their grades reported as points lost do better than if grades are reported as points gained and does not find any gender differences. While it is difficult to know how generalisable these results are given that each was carried out in different places, at different stages of the educational process, by different teachers, in different disciplines, the existing literature points to loss aversion as a plausible way in which teachers can increase student engagement and outcomes. Fry, Ketteridge, and Marshall (2009, p. 292) identify peer assessment as 'students assessing the work of others as a similar level'. Topping (1998, p. 250) in an extensive review of the literature defines peer assessment as 'an arrangement in which individuals consider the amount, level, value, worth, quality or success of learning . . . .of peers of similar status'. The literature advises us that peer assessment offers an opportunity to involve students in the assessment in a critical way. In Ireland, the National Forum called out assessment as their central enhancement theme for two years (2016-18). Some outputs emanating from research projects includes a National Forum insight report on students as partners in the assessment process. Included in their recommendations are to '[G]ive students opportunities to self-and/or peer-assess their work in a module'; and '[S]upport students to work with assessment criteria to monitor their own learning' (National Forum, 2016). Peer assessment is considered a powerful learning tool as it is ' . . . assessment for learning NOT assessment of learning'; despite this value, it is also ' . . . challenging with students who may not feel comfortable marking the work of their peers' (Wride, 2017, p. 4). Li, Liu, and Steckelberg (2010) recommend that anonymous peer assessment can help to reduce the discomfort students may feel at correcting the work of their peers. Supports and mechanisms need to be in place for students engaging in such a process. An assessment rubric is one such provision and will have the added benefit of helping to enhance student academic integrity, as outlined in twelve principles developed by the Teaching Enhancement Unit (2018) as part of a research project. Razi (2015) discusses the challenges associated with students writing an academic paper, and lecturers assessing it. In an effort to reduce the risk of plagiarism, he recommends the use of a detailed rubric designed by the lecturer who will use it, and aligned to the outcomes of the module.
The concept of loss aversion, when combined with peer assessment, can focus the students mind on using the detailed rubric in an in depth manner to grade a peers work. The motivation to do so stems from the risk of losing marks if they do not fully engage with the rubric when peer assessing.

Assessment process
Students were required to write only one essay which was graded by the lecturer out of 100% (part 1a). The grade was not revealed at this stage and was modified by their performance on the second element (part 1b). In part 1b, the students received at random the anonymous essay of a peer and were asked what grade they believe it deserved. This peer assessment did not influence the grade of the essay they were assessing, but for every 4% difference between their assessment and the grade the lecturer gave the essay, the peer assessor lost 1% mark from the total they earned in the first stage (Table 1). A valid concern is that reducing grades that have already been granted could be unfair. It is therefore important to note that DCU students understand that all grades are provisional until the promulgation of exam results at the end of the semester. Students were essentially being graded on a bundle of assessments framed to elicit greater effort and engagement.
Students who did not meet a threshold of 45% in part 1a were obliged to write a second essay and could not participate in the peer assessment phase. In DCU, a passing grade is 40%. This restriction was imposed to limit the chances of a student barely passing the essay and then failing on account of the peer assessment. These students' final grade was an equally weighted average of both essays.
This Continuous Assessment (CA) was designed to tackle superficial engagement with essay writing and plagiarism concerns. The aim was that it would oblige the students to engage with the essay writing process in a more meaningful way and think about the elements of a good essay. The students had access to the grading rubric and were actively and continuously encouraged to keep this in mind when writing their own essay and subsequent evaluation of their peer's work. It also leveraged the concept of loss aversion (Kahneman & Tversky, 1979) whereby people put a higher weight on losses than on gains, to encourage them to think deeply and engagethey will work harder to avoid losing marks than they would to gain them.
Moodle, the university's VLE, was used to leverage the approach using the workshop activity and grading rubric. The grading rubric was initially developed as a holistic rubric across five criteria (Table 2).
These criteria were advanced to an analytic rubric by providing descriptors for criteria, thus helping students understand the detailed expectations required for each criteria. This facilitated students to self-and peer-assess in a more targeted way and facilitated more reliable grading. The analytic rubric was imported to the grading rubric in Moodle. This rubric was used as part of Moodle's workshop activityan advanced peer review/ assessment activity which empowers learners. However, it is complicated with many options to set to configure the workshop. Therefore, it needs thoughtful planning and clear explanations to participants. The workshop activity allowed students to evaluate their own essay before submission (self-assessing) through the predesigned and defined analytic rubric. After submitting their essay, they then evaluated their (randomly selected) peer's work anonymously according to the same rubric. The workshop activity has four different phases (see Figure 1). In the setup phase, students were provided with instructions for submission, the rubric, and deadlines for the submission and assessment phases along with instructions for the peer evaluation in the assessment settings. In the submission phase, they viewed assessment instructions, the analytic rubric, and submitted their essay. Upon submission of all essays (or the submission phase deadline) the lecturer assessed them using the rubric. Students could not see any mark nor feedback given to their essays at this phase. Once the lecturer finished assessing all submitted essays, he informed those who did not qualify to complete 1b of the assessment phase and then manually allocated a random essay to eligible students. The grading evaluation phase involved the final mark being calculated by comparing the peer assessor's and lecturer's mark for the same submission ( Figure 2). As discussed above, for every 4% difference between the peer assessor mark and the lecturer mark given for the same essay, the peer assessor loses 1% mark from the mark they earned for the essay they submitted.  The final phase was to close the activity. When the workshop closed, students checked the overall mark in the Gradebook and any lecturer feedback.
Much of the success of the workshop activity depended on the students' understanding of the peer assessment criteria and their ability to correctly implement them. Therefore, it was essential that the assessment criteria (analytic rubric) were clear and easy to understand.
In recognition of the challenge associated with students engaging in peer assessment, the authors included information sessions at the beginning of two classes to induct students into the peer assessment process. One session involved demonstrating the technological aspects of the assessment, and the other teased out application and understanding of the detailed rubric, as recommended by Wride (2017). Additionally, the lecturer regularly reminded students to use his weekly office hours to help them with the peer assessment component.

Methodology
We received ethical approval to conduct this study. A mixed-methods approach was adopted as '[I]t is claimed that MMR (Mixed Methods Research) enables a more comprehensive and complete understanding of phenomena' (Cohen, Manion, & Morrison, 2018, p. 33). Data was collected immediately after the peer review element was completed and before the grades were published, using an anonymous survey via Google forms and two in-depth interviews with four students. Students were issued with a plain language information statement and their consent was required for both survey and interview (tick box for survey and signed consent letter for interviews). The survey was used because it offers a ' . . . cheap, reliable, valid, quick and easy . . . [and] is a widely used and useful instrument for collecting survey information' (Cohen et al., 2018, p. 471); The interviews allowed us to explore the assessment experience with participants in greater depth, and the questions used were open-ended to enable deeper investigation of students views and opinions (Cohen et al., 2018;Creswell, 2013). Both the survey and interviews were conducted prior to the students being made aware of their grades, though those taking part 2b were aware that they had not made the cut-off of 45%.
The survey contained several statements and students were asked to rate their agreement or not across a Likert five scale rating system. The statements solicited student opinions about the course, the assessment, the rubric, and skill development. Out of the 116 students who submitted work, 48 (41%) responded to the survey. Given the nature of our anonymous survey, we do not have the opportunity to link attitudes to the assessment with measures of student performance or socio-economic and cultural variables.
The in-depth interviews were used to provide deeper exploration about students experience and perception of the assessment strategy. Transcripts from the interviews were analysed using Braun and Clarke (2006) six phases of thematic analysis. They describe thematic analysis as ' . . . a method for identifying, analysing, and reporting patterns (themes) within data ' (2006, p. 6). The phases involved include becoming familiar with the data which was achieved through transcription of recorded interviews; generation of initial codes; and development of themes -NVivo was used for these two phases. The themes were reviewed (phase four) and refined (phase five) through discussions by the three authors in relation to the statistical analysis. The final phase is reporting on the findings.
Limitations in this study include the small number of students (n.4) who volunteered for the interview, and the fact that the data from the survey had not been analysed at that stage. It would have been interesting to explore some of the data findings. Additionally, a missed opportunity may be in the fact that students did not offer any peer feedback and so the learning for students was only in reviewing a peer's work, and not in receiving formative peer feedback. Boud (2000, p. 157) cautions that the use of grades alone can act as a barrier to student learner.

Attitudes to module and assessment
The majority of students either agreed or strongly agreed that they found the module interesting (75%) and useful (63%). The two opinions were strongly correlated (0.76). However, only 40% agreed or strongly agreed that they liked the way in which the module was assessed as opposed to 31% who disagreed. Even clearer evidence of an aversion to loss, or at least a system designed to elicit loss aversion, comes from the 60% who strongly agreed and 19% who agreed with the statement that they would have preferred a system in which marks can only be gained and not lost. However, only 36% would have preferred to write two essays.
Thirty-eight percent thought the way in which the module was assessed was unfair versus 33% who found it fair. Not surprisingly, finding the assessment to be fair was strongly correlated with liking the method (0.71). The correlation between viewing the assessment as fair was less strongly correlated with the view that the module was interesting (0.44) and useful (0.54). Clearly it was possible to find the assessment discomforting but still find the module engaging and valuable.

Student engagement and development
In terms of student engagement, the results also suggest that the intentionally fostered aversion to losing marks was not a barrier. More than half (56%) of respondents agreed that the assessment approach engaged them with only a little over a quarter disagreeing (27%). A little over half (51%) felt that they were involved in the assessment process for the module as evident in the following sample excerpts -'I felt a lot more in control of my own grade'; and 'I was a bit frustrated, but I did feel more involved'.
The most important finding from our survey, and the one that validates our approach is that 75% agreed or strongly agreed that 'The approach encouraged me to think critically and more deeply about what constitutes a good essay.' The remaining 25% was equally split between contrary and neutral views. Given that previous cohorts had had issues with referencing and plagiarism, it is also instructive to note that 71% agreed or strongly agreed that the nature of the assessment encouraged them to pay more attention to referencing and citing, with comments like 'I don't think I've ever used that many sources in an assignment before'. Finally, 60% felt the assessment helped them to develop their essay writing skills. It is likely that the rubric contributed to these views as 65% agreed or strongly agreed that using the rubric assisted them in evaluating their work before submission.

Student outcomes
While our survey strongly suggests that students benefited from the course and the majority enjoyed it, it is essential to have some objective evidence that student outcomes were not harmed by this method of assessment. Figure 3 shows that two-thirds of students 'lost' nothing or only 1% from the grade they earned for their own essay on account of the peer assessment. This translates to 1% of one-twelfth of their overall degree grade. Given that this method of assessment obviated the need for a second 3000word essay, one would like to think that students interested in grade maximisation would have devoted the saved time to assessment for other modules and in that way realised a net gain. An additional take away from this is that students, armed with a good rubric, are capable of reasonably accurate assessment of their peers' work.

Conclusions
Our findings suggest that this novel assessment strategy facilitated engagement and academic writing skills development by involving students in a peer assessment exercise and using detailed rubrics. In particular, students felt their critical thinking skills were enhanced as a result of this approach. They welcomed the guidance the rubric offered them and the opportunity to in some way gauge their performance against a peer's work. Equally they were motivated to perform by the dual-risk of potentially losing marks and the possibility of having to do two essays. Using the VLE workshop and rubric activities facilitated anonymous peer-reviewing and assisted in terms of efficiency from the lecturer's perspective. Although it must be acknowledged that the preparation time for the first iteration of this approach was significant. We expect this investment will reduce in subsequent iterations.
One of the most notable recommendations to emanate from this study was the need to provide support to students through detailed rubrics; information sessions on how the assessment strategy worked and was managed; and the lecturer being available and approachable during weekly office hours. Additionally, an innovative assessment approach, such as this one, requires a lot of upfront planning and preparation.
Our results show that by leveraging insights from behavioural economics, it is possible to design assessment that is simultaneously efficient in terms of teacher effort and effective in terms of fostering engagement and critical thinking. Further research in this vein could examine if such approaches are more or less effective with different types of student and module.