Handing in a paper or exercise and merely receiving "bad" or "incorrect" as feedback is not very helpful when the goal is to improve. Unfortunately, this is currently the kind of feedback given by many Automatic Short Answer Grading (ASAG) systems. One of the reasons for this is a lack of content-focused elaborated feedback datasets. To encourage research on explainable and understandable feedback systems, we present the Short Answer Feedback dataset (SAF). Similar to other ASAG datasets, SAF contains learner responses and reference answers to German and English questions. However, instead of only assigning a label or score to the learners' answers, SAF also contains elaborated feedback explaining the given score. Thus, SAF enables supervised training of models that grade answers and explain where and why mistakes were made. This paper discusses the need for enhanced feedback models in real-world pedagogical scenarios, describes the dataset annotation process, gives a comprehensive analysis of SAF, and provides T5-based baselines for future comparison. 1 1 Introduction Assessment and feedback are essential to high-quality education (Shute, 2008). They allow learners and teachers to discover misconceptions, gaps in knowledge, and improvement opportunities. However, manually assessing learners' knowledge and providing helpful feedback is time-consuming and requires pedagogical as well as domain expertise. Here, automatic assessment can free up teachers' time to focus on tutoring learners or adequately preparing classroom activities. Moreover, it can be an alternative to peer-grading when course participant numbers increase beyond the financial 1 Our code, scoring rubrics and dataset are available at https://github.com/SebOchs/SAF under an MIT license feasibility of manual grading (Kay et al., 2013), making it particularly useful for freely accessible online courses. Besides being costand time-efficient, automating assessment also offers unique teaching opportunities. As long as systems give individual, response-specific feedback, learners may retry or take additional assignments and receive instantaneous feedback as often as they need. Additionally, knowing that a system instead of one's teacher or professor will evaluate one's assignment can also reduce anxiety and help learners focus on their work instead of worrying about their reputation (Lipnevich and Smith, 2009). Therefore, it is unsurprising that automatic assessment has been an active research field over the past decades (Burrows et al., 2015; Ihantola et al., 2010; Ke and Ng, 2019; Xi, 2010). So far, significant progress has been made. In particular, Transformer models are approaching human experts' performance on specific datasets in the Automatic Short Answer Grading (ASAG) field (Sung et al., 2019; Camus and Filighera, 2020). These models are trained to evaluate whether natural language responses fully answer open knowledge questions and typically output a score or label indicating the response's correctness. This kind of feedback is also called verification (Shute, 2008). An example can be seen in Table 1. However, merely providing a score or label for a learner's answer is generally not sufficient in real-world pedagogical scenarios. Firstly, learners must understand their feedback to use it effectively (Winstone et al., 2017). That may not be the case when learners only receive a score instead of a clear explanation of where and why they made mistakes. Secondly, the feedback's source needs to be trusted for learners to accept and engage with the given advice (Winstone et al., 2017). Especially assessments by automatic models may be questioned (Lipnevich and Smith, 2009; Filighera et al., 2020a,b). Providing a response-specific, detailed 8577 Question: What are the challenges of Mobile Routing compared to routing in fixed and wired networks? explanation may establish the necessary trust in the system's predictions. This kind of explanation is also called elaborated feedback (Shute, 2008) and is shown in Table 1. In the Intelligent Tutoring Systems community, the need for elaborated feedback is well-known (Deeva et al., 2021; Hasan et al., 2020). Several researchers have incorporated feedback modules in their systems (VanLehn, 2011; Kulik and Fletcher, 2016; Mousavinasab et al., 2021). However, these approaches are typically constrained to structured answer formats, such as programming exercises (Keuning et al., 2018), focus on the response's language and style instead of the content (Hell-man et al., 2020), or are hand-tailored to specific tasks (Dzikovska et al., 2014; Lu et al., 2008). A lack of public, content-centered elaborated feedback datasets may be one of the main reasons for these limitations. To narrow this gap, we provide the Short Answer Feedback dataset (SAF), a German and English collection of learner answers and feedback. In contrast to other ASAG datasets, SAF contains detailed elaborated feedback explaining the scores assigned to learner responses. This allows for automatic scoring and opens the new task of providing response-specific, elaborated feedback illustrating a given score. The dataset currently contains 4,519 submissions, corresponding scores, and response-specific elaborated feedback. Additionally, we provide T5 (Raffel et al., 2020) and mT5 (Xue et al., 2021) baselines for future comparison. While elaborated feedback datasets on language learning (Caines et al., 2020; Pilan et al., 2020; Stasaski et al., 2020) appeared recently, they focus on linguistic mistakes, such as grammatical errors, instead of content. Our extensive literature review did not reveal datasets that included content-focused elaborated feedback on short answer responses. However, SAF's feedback can be viewed as a textual explanation of the assigned score. Therefore, comparable NLP datasets with textual explanations and publicly available ASAG datasets without explanations are discussed in the following sections. In recent years, the need for understandable, interpretable NLP models has been widely discussed (Adadi and Berrada, 2018; Alishahi et al., 2019; Danilevsky et al., 2020; Das and Rad, 2020). One of the possible approaches to make models explainable is to train them or auxiliary models to directly generate explanations of their predictions (Liu et al., 2019; Narang et al., 2020). For this purpose, multiple researchers enhanced NLP datasets with textual explanations. Camburu et al. (2018) extended the Stanford Natural Language Inference dataset (SNLI) (Bow-man et al., 2015) using Amazon Mechanical Turk. The expanded dataset is called e-SNLI and contains textual, human-generated explanations for each of SNLI's entailment relation pairs. Ra-jani et al. (2019), also using Amazon Mechanical Turk, expanded COMMONSENSEQA (Talmor et al., 2019). The resulting Common Sense Explanations (CoS-E) dataset consists of commonsense reasoning questions with three possible answers and a textual explanation for every correct selection. Mostafazadeh et al. (2020) introduced 8578 GLUCOSE , a crowdsourced collection of semi-structured causal explanations related to sentences in stories. However, the datasets above do not have a pedagogical focus. This is detrimental to researchers aiming to employ their systems in educational contexts, where explanations should conform to pedagogical guidelines, such as avoiding harm to the learner's self-esteem or motivation. The closest to our research is the WorldTree V2 dataset. Here, Xie et al. (2020) used graphs of expert-engineered natural language facts to explain correct answers to multiple-choice science questions. The resulting explanations are essentially lists of scientific and world knowledge facts needed to answer the question correctly. Similarly, Ling et al. (2017) provide textual explanations for the correct solutions to math problems. Their multiple-choice questions, answers, and explanations are obtained by crowdsourcing and standardized tests, such as GMAT. While both Ling et al. (2017)'s and Xie et al. (2020)'s work have an educational focus, they only explain the reference solution instead of mistakes made in incorrect or partially correct solutions. Some of the most well-known ASAG datasets stem from the SemEval 2013 challenge (Dzikovska et al., 2013). BEETLE contains 5,044 student answers to basic electricity questions labeled as correct, partially_correct_incomplete, contradictory, irrelevant or non_domain . SCIENTSBANK follows the same structure but also contains questions of various other domains, such as biology or geography. Basu et al. (2013) introduced Powergrading , a collection of 2,532 unique, crowdsourced answers to ten questions of a United States Citizenship Exam. Each was manually classified as correct or incorrect . In contrast to the previous datasets, answers in the ASAP-SAS 2 dataset are scored on a scale from 0 to 3. Additionally, this dataset is much larger with  2,200 responses per question, with 10 questions in total. All of the datasets above only include verification feedback. Mizumoto et al. (2019) released a Japanese dataset containing 12,600 student responses equally distributed across 6 questions. The answers stem from a commercial achievement test for Japanese high school learners and are annotated with holistic scores and individual marks for manually defined 2 https://www.kaggle.com/c/asap-sas/ scoring criteria. Additionally, each criterion links to the phrase in the student's answer expressing it. For example, for a criterion like "2 points if the response mentions Western culture ", the phrase Western culture would be marked in the response, if present. This dataset enables elaborated feedback systems. However, the structured nature of criteria and matching answer spans complicates an automatic translation to English. Additionally, the marking scheme is limited in its expressiveness as it is hard to mark missing information in the answer. Lastly, structured collections of smaller and nonpublic datasets can be found in surveys by Roy et al. (2015) and Burrows et al. (2015). To remedy the lack of content-focused elaborated feedback datasets, we provide SAF, an English and German short answer dataset with explanations that serve as elaborated feedback. In total, the corpus contains 4,519 submissions similar to the example in Table 1. There are 22 English short answer questions with reference answers covering a range of college-level communication network topics, such as extension headers in IPv6 or frame bursting. Additionally, the dataset contains 8 German short answer questions used in micro-job training on the appJobber 3 crowd-worker platform. The data was collected and annotated between April 2020 and June 2021. While individuals gave the German answers in the context of pre-job training, the English questions were answered in groups of up to three students in voluntary quizzes they could complete for extra points in the final exam. Each quiz consists of 3-4 questions regarding the same overarching topic, such as Internet protocols. All answers are annotated with a score, label, and feedback as described in Table 2. The dataset can be used for classical automatic short answer grading and elaborated feedback generation. We need reliable scoring and clear, detailed explanations to train understandable feedback models. Providing this is challenging for multiple reasons. Firstly, annotators need to have the necessary domain expertise and the pedagogical knowledge on how to provide understandable, well-received feedback. For instance, they should be aware of their 3 https://appjobber.de/ 8579 Field Description Score A numerical value between 0 and 1 indicating the answer's correctness and completeness. feedback's emotional effect. At first glance, this may seem obvious, but it is easily overlooked in practice. An example of this became apparent during a pilot study we conducted to uncover pitfalls and train our annotators. Even though we provided guidelines on how to give feedback, questionable phrases like "This response fails to ..." were common as the annotators did not consider that the word "failing" may trigger negative associations and emotions in learners. Secondly, a common ground truth must be established for each question with clearly defined boundaries because various sources may define concepts differently. For example, the network protocol TCP alone has at least five different variations, all with unique advantages and disadvantages, leading to multiple possible answers to TCP related questions (Chaudhary and Kumar, 2017). In our pilot study, this expressed itself with a low inter-annotator agreement (Krippendorff's Alpha of 0.36), making the need for detailed scoring rubrics clear. We discuss our approaches to these challenges in the following section. To ensure the necessary domain expertise, we selected two graduate students 4 who had completed the communication networks course themselves and two experienced appJobber employees for the crowd-worker platform's answers. For pedagogical training, a researcher first drafted a general annotation guideline . It explains the annotation files' structure, the annotation goals, and provides general recommendations for the formulation of feedback and the calculation of scores. For example, it asserts that praise, comparisons with other 4 The students' remuneration consisted of a paid research assistant position for one and partial credit towards a master's thesis and co-authorship of this paper for the other. learners, or emotionally charged words like "fail" should be avoided when writing feedback. Additionally, it points out common biases annotators should be aware of, such as confirmation bias. For instance, answers that contain keywords found in many correct responses may still contain mistakes and should, therefore, still be carefully inspected. The general annotation guidelines were submitted to a psychology doctoral student with prior work in the feedback field for additional advice. Then the annotators applied their knowledge in the pilot study and received further feedback from the researchers. Finally, the guideline was updated to reflect any additional discussion points. As can be seen in Figure 1, the researcher drafted grading rubrics for each question. The rubric consists of the questions, reference answers with detailed grading information, and four example answers per question for illustration. As research suggests that a single author may not suffice to produce reliable and objective scoring rubrics (Carr, 2020), the draft is then discussed and refined with the annotators. The discussion also mitigates the challenge of defining a common ground truth, as multiple sources and opinions can coalesce into a single, exhaustive rubric. Before the discussion, the answer annotation files are available to the annotators. The files contain the reference and students' answers. Subsequently, annotators individually evaluated answers using the scoring rubric and the general annotation guideline. All English answers were annotated twice, while only half of the German answers were annotated doubly due to the prohibitive cost of experienced employees. The first step of combining the independently annotated answer files into a cohesive gold standard involved discussing disagreements with the annotators and researcher. Disagreements between the annotators 8580 Researcher Final GradingRubric GradingRubric Draft AnswerAnnotationFiles Annotators D i s c u ss i on GradedAnswers & Feedback Annotator 1 Annotator 2 GeneralAnnotationGuidlines GradedAnswers & Feedback Discussion of Disagreements Grammar and Spell Checking Gold Standard Guideline Generation Annotation Researcher Figure 1: Schematic depiction of the annotation process. were resolved by either choosing one of the annotations, compromising, or fusing them if both had merit. For example, one annotator may notice a missing fact A while the second annotator may find a mistake in B's explanation. Finally, the English gold feedback was checked by Grammarly as well as an English native speaker. Grammar and spelling mistakes were corrected, and sentences were simplified when the same information could be expressed more concisely, for example, by using the possessive form. Learners' answers were not post-processed because models would frequently encounter grammar and spelling mistakes in the wild. Therefore, this is a challenge approaches should overcome. The annotation process resulted in a corpus with the following score and label distribution seen in Table 3. Similar to the SemEval dataset BEETLE (Dzikovska et al., 2013), we split the data into training (64% of DE / 70% of EN), unseen answers (11% / 12%) and unseen questions (25% / 18%) test sets. While the unseen answers test split contains new answers to the training's questions, the unseen questions split contains novel questions. This setup enables the investigation of models' ability to generalize to new questions without the need for priming with manually annotated answers first. Figure 2 shows the length of questions, feedback, reference, and learner answers of the English training set in tokens. We used NLTK's word_tokenize 5 to obtain the tokens, so their count 5 https://www.nltk.org/api/nltk. tokenize.html Score Train UA UQ DE EN DE EN DE EN 0.0 216 234 47 42 49 87 (0 . can be seen as the sum of words and punctuation symbols in the text. The learners' answers were between 0 and 589 tokens long (average=82.2, me-dian=68). We did not filter empty submissions (unless all of the group's submissions were empty) from the dataset as models will encounter this in real-world applications. Since the reference answer and learner answer are typically combined as input for ASAG models, this dataset's sensible input sequence length may prove to be computationally expensive for large Transformer models. Feedback tends to be shorter with 5-120 tokens (average=22.4, median=15). The distribution looks similar for the German half of the dataset only that the answers and feedback tend to be slightly shorter. Details can be found in Appendix A. 3.4 Annotation Quality To estimate our annotations' reliability, we rely on inter-annotator agreement measures. As the scores are interval scaled between 0 and 1, we report the 8581 10 20 30 40 50 60 70 80 0 2 4 6 8 10 C o un t percentage agreement and Krippendorff's Alpha. The annotators agreed on 89.46% of the cases on the English data, and  is 0.91 (N=2,112). On the German questions, the annotators agreed in 81.38% of the cases, and  is 0.78 (N=1,200). The high agreement on the overall dataset illustrates the effectiveness of our annotation process, especially when compared to the initially low agreement of  =0.36 achieved in our pilot study. We can assume the validity of our German data to be high, since our experienced annotators were also responsible for accepting or rejecting job results later on. Hence, their judgements should be consistent with the desired learning outcome. To estimate the validity of our English data, we assume that the end-of-term exam is a valid evaluation of students' knowledge. Of course, this is most likely not accurate in practice since the exam was not formally validated and only provides a snapshot of students' performance in a single 120-minute time frame. However, most of the question pool and exam structure have been employed and refined over multiple years. For this reason, we deem it a sufficient approximation. Nevertheless, the following results should be viewed as an indication of validity rather than a fact. The Spearman's rank correlation between the points achieved in the exam and the quizzes is 0.438 ( p < 0 . 0001 ) with a sample size of 186. This is a moderate positive correlation between the exam and quiz results (Dancey and Reidy, 2007) and indicates that they may measure the same or a similar construct. In contrast to the quizzes, exams were not taken in groups, partly explaining the variance. It is our responsibility to be transparent in our data collection process and protect the privacy of our learners. Our first step in this regard was to inform our learners of the data collection process. We posted to the college course's online learning platform and the description of the German job training. Both channels usually carry vital information for the learners. In our post, we  detailed how we would use the learners' answers to research and develop automatic assessment models. asked learners to refrain from including personal information in their answers, such as names or addresses. This was also checked during the annotation process. gave them contact information if they wanted their answers to be excluded from the data collection. We also clarified that this would not negatively impact them or their grades/access to jobs. None of the learners contacted us. clarified that we would only release anonymized data in our publications. We anonymized German answers by stripping identifying information and randomizing the order. To anonymize the English learners' answers, we randomly assigned each group an ID. The group-to-ID mapping was done locally on one computer and was deleted after the dataset construction. Keeping a consistent group ID allows us to identify responses with quizID.questionID.groupID and, thus, publish a dataset where the other answers of a group can be incorporated to refine an assessment model. For example, responses QuizA.1.3 and QuizB.2.3 are written by the group assigned the ID 3. This characteristic is beneficial as it allows for training models that provide personalized feedback, considering the current answer and answers to related questions. Patterns of mistakes spanning multiple questions may be discovered in this setting. For example, if a group answered all performance evaluation questions incorrectly, they may not understand the probability theory underlying the questions. However, note that SAF's an-8582 notators only considered the current answer when constructing their feedback. The goals of our experiments are threefold. Firstly, we want to provide baselines for the dataset. For this reason, it makes sense to report a wide range of metrics future work may want to utilize. Secondly, we hypothesize that including the question in the model's input would increase performance. Typically, only the student and reference answers are compared in ASAG (Lv et al., 2021) even though the question may contain additional important information. To investigate the question's effect on performance, we run each experiment in two settings: with a student and reference answer pair as model input or with a question, student, and reference answer triplet. Finally, we want to explore the synergy between the ASAG scoring and classification tasks and feedback generation. We believe that grading and feedback should be trained jointly since the feedback should match the assigned grade (Wiegreffe et al., 2021), and both tasks benefit from extracting the same information from the answers. For example, a span of tokens negatively impacting the grade should also affect the feedback accordingly. Our experiments investigate the hypothesis that feedback generation benefits more from being paired with the more informative ASAG scoring task (0-1) than the verification feedback label classification (correct vs. incorrect vs. partially correct). As baselines, we utilize HuggingFace's implementation of the T5-base and mT5-base models (Wolf et al., 2020). They are fine-tuned to predict the response's score or label and jointly explain it. For computational reasons, the input sequence is trimmed to 512 tokens when using T5 and 256 tokens when using mT5. When the sequence is longer, a part of the reference answer is truncated. While the complete learner answer is always relevant for grading, the reference answer may discuss details or additional aspects irrelevant to the particular response. The output is limited to 128 tokens and has the following format: " label/score feedback: feedback ". We also enforce a minimum output sequence length of 11 tokens since models tended to refrain from generating feedback otherwise. In all experiments, 10% of the training data was split-off for manual hyperparameter tuning and model selection. All models use gradient accumulation and an Adafactor (Shazeer and Stern, 2018) optimizer with learning rate warm-up. We trained models for maximally 64 epochs utilizing early stopping with a patience of 10 and selected the best performing model/epoch using the following metric m , where f is the macro-averaged F1 score during classification and 1  MSE during scoring. We average SACREBLEU, 6 ROUGE-2 and METEOR to compensate for the individual metrics' weaknesses when measuring the generated feedback's quality (Post, 2018; Banerjee and Lavie, 2005). Thus, m balances the feedback generation and labelling performance, such that success on both tasks is required. Each model trained for approximately 1-5 hours on 2 Nvidia RTX 2080 Ti cards with 11 GB of RAM. The mT5 models were trained on a single card, due to the memory overhead of parallelization. Table 4 shows T5's, a majority baseline's and the average human performance on the English test sets. The majority baseline predicts the most common label/score in the training set, paired with the most common corresponding feedback. In both datasets, the majority class consists of entirely correct responses. In German, the most common matching feedback is Korrekt! and in English, The response answers the differences cor-rectly. is predicted. We report the accuracy and macro-averaged F1 score for classification and the root-mean-squared-error for scoring. Additionally, we compare the generated and annotated feedback to the gold standard using BERTScore 7 (Zhang et al., 2020) in addition to the metrics used during validation. We can see that T5 provides a strong baseline for this task, outperforming the majority baseline significantly. However, there is still room for improvement compared to human performance, especially on unseen questions. A closer inspection of the generated feedback also revealed that the 6 https://pypi.org/project/sacrebleu/1. model would often, and often senselessly, copy common phrases it saw during training with mi-nor modifications (see Appendix B). This indicates that elaborated feedback tasks can be challenging even to large language models. Simultaneously, the models' high text similarity scores indicate a need for new evaluation metrics that measure similarity on a contentinstead of lexical-level, enforcing that a text not only sounds well but also makes sense. Contrary to our belief, providing the model with more detailed scores instead of only labels during training does not improve the feedback generation's performance. It even worsens performance slightly for most metrics. On the English data, we observed that the question provided only a marginal benefit for unseen answers and a larger benefit for unseen questions. Interestingly, this trend does not seem to extend to the German dataset, as depicted in Table 5, indicating that this effect may be language or dataset dependent. Additionally, we can see that generalizing to new questions is even less successful on the German than on the English data. This may be due to the distribution of questions and answers in the datasets. While both are of similar size, there are significantly fewer German questions with more answers per question than English ones. The divergent answers to questions ratio may also explain why mT5 on the German data outperforms T5 on the English data when classifying or scoring unseen answers. This paper introduces the elaborated feedback generation task. We provide a benchmarking dataset containing short answers, scores, and textual explanations of given scores to kick off this task. As of yet, the dataset consists of 4,519 submissions to German and English questions. We demonstrate SAF's reliability with high inter-annotator agreements. In Section 3.3, we presented aspects of the dataset we plan to improve. While the dataset is sizable for a manually annotated task of this complexity, it is small compared to other NLP tasks' crawled, large-scale datasets. We plan to mitigate this by incorporating additional questions in future iterations of the dataset. The focus will be on more complex questions to improve the class balance and questions of other domains and languages to increase diversity. The model's ability to general-8584 ize to unseen questions may also benefit from a more diverse dataset. We also observed that common text similarity metrics can provide a valuable first impression of the feedback's quality but are not sufficient to fully capture it. Thus, we would recommend including humans in the evaluation loop. A possible evaluation setup could ask annotators whether the generated feedback expresses the same meaning as the reference feedback included in the dataset. We believe annotators could also carry out this task with limited background in the provided domains. Nevertheless, we provide the detailed scoring rubrics utilized by our annotators along with the dataset to support future human evaluations. Finally, the baselines presented in this paper can be improved. Considering the deep understanding human graders require for this task, we believe neuro-symbolic approaches to be an exciting avenue of future research. Current models may especially benefit from incorporating knowledge bases and other reference material. We would like to thank the wer denkt was GmbH for their cooperation in the German data collection, our annotators for their hard work and dedication and Viktor Pfanschilling for his feedback and support. This research is funded by the Bundesminis-terium fr Bildung und Forschung in the project: Software Campus 2.0 (ZN 01|S17050), Micropro-ject: DA-VBB.
abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method other other abstain other other other other other other other other other other other other abstain other other other other other other other other other other other other other other other other other other other other method abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain method abstain objective result abstain method abstain abstain objective method abstain method method result method abstain abstain abstain