Answers.Body: <p>Unless the objective of the exercise is to copy an implementation (say from Java to C++), the "similarity" to any other implementation should have no weight in the grading. At least not in a positive sense. The similarity to another implementation could, however, be a warning sign for plagiarism. Everything else in the question is <em>up for debate</em>, so to speak.</p>
  
  <p>The "fairness" of the grading only hangs on two principles: Consistent application of standards and advance notice of those standards. If the same standard, or evaluation, is applied to all the work received, and the assigned grade would be the same regardless of which student turned it in, or which TA assigned the grade, then the standards are applied consistently. Prior to the examination, or assignment, the students need to be informed as to how their work will be evaluated. In the course syllabus, or in the beginning of the course is the best time to inform them, of course. If they have been given such a notice, and then their work is evaluated according to that guide, there should be nothing to object to - from a "fairness" point of view. Knowing that there will be unit tests, even without knowing what those might be, is still acceptable, so long as the same unit tests are applied in a uniform manner to all the work evaluated.</p>
  
  <p>How "fair" the evaluations are, however, likely has very small significance to how well the evaluations move the students toward the course objectives. Likewise, how such projects would be evaluated in a real-world workplace environment likely does not move them in the direction of meeting the course objectives. Of course, if the class is the <strong>final</strong> course in a series, the objective of which is to produce work-ready employees, then a purely external view of <em>works</em> vs. <em>fails</em> might be appropriate. In all other cases, however, such an extreme approach is not a very pedagogical approach. (As a side note, a binary go/no-go grade does not fit well into a scaled grading system either.)</p>
  
  <p>In an idealized world, the student projects are a chance for the instructor to determine how well the student has apprehended the course materials thus far, discover where the student has stumbled, and take corrective measures to return the student to where they should be relative to the course materials. Teachers, of every stripe, are an over-worked, under-paid, unappreciated lot. (The TA's get to share in that "glory", and will become just as over-worked as the instructors.) As such many of the opportunities to learn about the difficulties and misunderstandings the students have will be missed when grading assignments. Having a rubric, published by preference, which the TA can apply to the evaluation often helps. </p>
  
  <p>The course objectives, naturally, have greater weight in the rubric than other criteria. Yet non-course objectives should be included. While seldom included in the course specifications, clean code, naming conventions, documentation or self-documenting code, and adherence to coding style can all be covered by the rubric, and encouraged, thereby, in the students' work. Requiring adherence to such guidelines benefits the instructor (and TA's) in applying fine-grained grading evaluations. It can also aid the students in completing the projects, obtaining better results from the code, ease the burden of debugging the code prior to submission, and developing good habits for when they do enter the workforce, where go/no-go evaluations really matter. (I prefer to give more points to a clean almost-working program, than to a impenetrable kludge that passes every known test. The former can be maintained, including fixing the current version. The latter can probably only be maintained after being re-coded in some understandable manner.)</p>
  
  <p>Unfortunately, my view is only one way of seeing the job of "instructor." As to <em>it is okay to grade programming assignments based on this external view</em>, the answer is; "While it is less than ideal, yes, it is okay to do so."</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-12T01:58:37.290Z
Answers.LastActivityDate: 2018-12-12T01:58:37.290Z
Answers.UserDisplayName: Gypsy Spellweaver
Answers.Body: <p>Actually, life isn't so simple as to choose one <em>or</em> the other here. Code needs to be well structured <em>and</em> it needs to be correct. Given that students are learning, you can't really expect that they will be perfect on either of these and so you need to take both in to account. A program could, be well designed and built but have some, as yet undiscovered, flaw that prevents it from passing its tests. If the students hack it up to get tests to pass they are going backwards, not forwards. </p>
  
  <p>CS students have a lot to learn. Programming is important, but not the most important thing. But to advance on many of the other goals of the CS curriculum, students need to build good code. Even better, would be the ability to build maintainable code, since many of them will start their careers in programming-heavy jobs. Even if they go to grad school, much of what they do will depend on being able to write good code. </p>
  
  <p>So, in grading, points should be gained or lost depending on both code and correctness. If the code is not correct, then someone needs to make an analysis about why the code is failing. It could be something simple or profound. </p>
  
  <p>In my view if a program can't be understood easily and/or seems to be hacked together, it should receive few points. If a program, on the other hand, passes few tests then it is a much more subtle problem that needs deeper analysis. Don't neglect it. </p>
  
  <p>In the question heading you talk about the "correct implementation". I worry about that. Students should not be graded on similarity to any reference implementation, certainly. There are more than a few ways to write a correct program. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-11T23:36:51.640Z
Answers.LastActivityDate: 2018-12-11T23:36:51.640Z
Answers.UserDisplayName: Buffy
Answers.Body: <p>I agree with Buffy, the question would be perfect for cseducators SE. But still, I try to give an answer which might be generalized to other disciplines as well:</p>
  
  <p>Of course, this is absolutely ok - but it affects your teaching style and goal.</p>
  
  <p>If you want the students to be 100% perfect, this is the way to go. It's a hard lesson, but in real world scenarios they are facing more or less the same situation. My Physcis teacher at school took the same attempt and argued, that it does not make a difference if you want to drive to school and crash into the tree next to school or next to your house - you don't make it!</p>
  
  <p>I fully understand the burden of reading students code - I do the same and it can be a great pain. But if you do it, you can give individual feedback and you'll understand, which are the most common mistatkes and adress them in your course. </p>
  
  <p>One compromise could be to offer different test sets, let's say some tests for each function needed for the complete task. That would allow to give points partially, but still you can not give points for a correct recursion, loop or whatever. </p>
  
  <p>An additional thought: You might consider giving a few test cases so the students can at least test if their code understands the input and delivers the output in a format your test understands. I don't know if this is a real world problem with your students, but it would be an issue with mine ;-). </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-11T21:49:59.073Z
Answers.LastActivityDate: 2018-12-11T21:49:59.073Z
Answers.UserDisplayName: OBu
Answers.Body: <p>The 'external view' is important as it ensures the submission is correct. These tests should be similar to Acceptance Testing since you are checking whether the submission meets the 'business' requirements i.e. the requirements laid out in your question. I highly recommend automating this process with unit tests or console based tests (check output matches the expected output).</p>
  
  <p>The 'internal view' is what motivates your students to do better since only a person can judge this properly. Students are not motivated to write good code when they are only being judged by automated tests. I recommend looking over each student's submission and ensuring that they have followed the correct approach have met proper code quality standards.</p>
  
  <p>Of course, you can't give a lot of time to looking over each submission UNLESS the 'external view' tests are automated and you don't have to download and run the code for the 'internal view' reviews.</p>
  
  <p>I suggest automating your process using <a href="https://autogradr.com" rel="nofollow noreferrer">AutoGradr</a> (disclaimer: I am the author of this free tool)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-14T16:27:37.870Z
Answers.LastActivityDate: 2019-05-14T16:27:37.870Z
Answers.UserDisplayName: tusharsoni
Body: <p>Imagine students in a course on algorithms &amp; data structures, which are given homework assignments to implement things like trees, hash sets, sorting algorithms, etc.</p>
  
  <p>Due to the high number of students per semester and limited resources, grading is heavily assisted by automatic unit tests. The TAs will still scan through the code, but if all or most of the test cases fail and there is no simple fix in sight, students will get 0 points on an assignment.</p>
  
  <p>Unfortunately, this can happen very quickly. Consider implementing a balanced tree as an example. If students fail to correctly reference the nodes, insertion, removal and searching, - despite being implemented correctly if seen individually - will not work and can hardly be tested.</p>
  
  <p>As expected, some students do complain about their poor grade and argue that <em>"although they acknowledge that their data structure does not work entirely, they at least implemented various sub-tasks correctly"</em>. This may even be correct, but the TAs very often do not have the time to dig deep in the code and grade the internal functions of the students implementation. Especially for more complex data structures, it would also take too much time to fully comprehend and even debug the (erroneous) implementations.</p>
  
  <hr>
  
  <p><em>My question is, if it is okay to grade programming assignments based on this <strong>external view</strong> (does the implementation pass the tests?) or should grading rather be based on an <strong>internal view</strong> (are the things implemented by the student correct if seen individually?).</em></p>
  
  <p>One important remark: In this specific scenario, students were not given the unit tests, since they are expected to test their implementation themselves.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-11T21:46:21.633Z
Comments.Text: Are they provided with an environment where they can run and test their code?
Comments.UserDisplayName: Acccumulation
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-11T22:00:07.157Z
Comments.Text: @ff524 In this situation, students are aware of the test cases. The actual grading follows a rough catalogue but is mostly up to the TAs. One more thing for clarification: I am a student representative in this case and generally thinking about an opinion on that matter.
Comments.UserDisplayName: blu3r4y
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-11T22:02:21.330Z
Comments.Text: @Acccumulation Yes, the students know about testing from previous courses and could easily write some along with the actual implementation.
Comments.UserDisplayName: blu3r4y
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-12T09:52:40.890Z
Comments.Text: Assignments should be graded based on the educational goals of the course. I can see value in courses that teach _writing code to fit a given specification_, and different value in courses that teach _implementing specific algorithms and data structures_. The same code submission should be graded differently in those two classes.
Comments.UserDisplayName: JeffE
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-18T12:02:31.067Z
Comments.Text: Publish the success criteria, and grade against it: publish the unit tests.
Comments.UserDisplayName: ctrl-alt-delor
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-13T16:28:00.553Z
Comments.Text: **Unit tests** should **test the units** of the program, not just the whole thing. Therefore students can write sub-parts and pass tests, and therefore get grades.
Comments.UserDisplayName: ctrl-alt-delor
Tags: <grading><homework>
Title: Should programming assignments be graded based on results from unit tests or based on the similarity to the correct implementation?
ThreadText: # Should programming assignments be graded based on results from unit tests or based on the similarity to the correct implementation?
  
  (Asked by: username_0 on 2018-12-11)
  
  Imagine students in a course on algorithms & data structures, which are given homework assignments to implement things like trees, hash sets, sorting algorithms, etc.
  
  Due to the high number of students per semester and limited resources, grading is heavily assisted by automatic unit tests. The TAs will still scan through the code, but if all or most of the test cases fail and there is no simple fix in sight, students will get 0 points on an assignment.
  
  Unfortunately, this can happen very quickly. Consider implementing a balanced tree as an example. If students fail to correctly reference the nodes, insertion, removal and searching, - despite being implemented correctly if seen individually - will not work and can hardly be tested.
  
  As expected, some students do complain about their poor grade and argue that *"although they acknowledge that their data structure does not work entirely, they at least implemented various sub-tasks correctly"*. This may even be correct, but the TAs very often do not have the time to dig deep in the code and grade the internal functions of the students implementation. Especially for more complex data structures, it would also take too much time to fully comprehend and even debug the (erroneous) implementations.
  
  ---
  
  *My question is, if it is okay to grade programming assignments based on this **external view** (does the implementation pass the tests?) or should grading rather be based on an **internal view** (are the things implemented by the student correct if seen individually?).*
  
  One important remark: In this specific scenario, students were not given the unit tests, since they are expected to test their implementation themselves.
  
  **Tags:** grading, homework
  
  > **Comments:**
  > **username_1**: Are they provided with an environment where they can run and test their code?  
  > **username_0**: @ff524 In this situation, students are aware of the test cases. The actual grading follows a rough catalogue but is mostly up to the TAs. One more thing for clarification: I am a student representative in this case and generally thinking about an opinion on that matter.  
  > **username_0**: @username_1 Yes, the students know about testing from previous courses and could easily write some along with the actual implementation.  
  > **username_2**: Assignments should be graded based on the educational goals of the course. I can see value in courses that teach _writing code to fit a given specification_, and different value in courses that teach _implementing specific algorithms and data structures_. The same code submission should be graded differently in those two classes.  
  > **username_3**: Publish the success criteria, and grade against it: publish the unit tests.  
  > **username_3**: **Unit tests** should **test the units** of the program, not just the whole thing. Therefore students can write sub-parts and pass tests, and therefore get grades.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_5 on 2018-12-11. Score: 7
  
  Actually, life isn't so simple as to choose one *or* the other here. Code needs to be well structured *and* it needs to be correct. Given that students are learning, you can't really expect that they will be perfect on either of these and so you need to take both in to account. A program could, be well designed and built but have some, as yet undiscovered, flaw that prevents it from passing its tests. If the students hack it up to get tests to pass they are going backwards, not forwards.
  
  CS students have a lot to learn. Programming is important, but not the most important thing. But to advance on many of the other goals of the CS curriculum, students need to build good code. Even better, would be the ability to build maintainable code, since many of them will start their careers in programming-heavy jobs. Even if they go to grad school, much of what they do will depend on being able to write good code.
  
  So, in grading, points should be gained or lost depending on both code and correctness. If the code is not correct, then someone needs to make an analysis about why the code is failing. It could be something simple or profound.
  
  In my view if a program can't be understood easily and/or seems to be hacked together, it should receive few points. If a program, on the other hand, passes few tests then it is a much more subtle problem that needs deeper analysis. Don't neglect it.
  
  In the question heading you talk about the "correct implementation". I worry about that. Students should not be graded on similarity to any reference implementation, certainly. There are more than a few ways to write a correct program.
  
  
  ---
  
  ### Answer by username_6 on 2018-12-11. Score: 3
  
  I agree with username_5, the question would be perfect for cseducators SE. But still, I try to give an answer which might be generalized to other disciplines as well:
  
  Of course, this is absolutely ok - but it affects your teaching style and goal.
  
  If you want the students to be 100% perfect, this is the way to go. It's a hard lesson, but in real world scenarios they are facing more or less the same situation. My Physcis teacher at school took the same attempt and argued, that it does not make a difference if you want to drive to school and crash into the tree next to school or next to your house - you don't make it!
  
  I fully understand the burden of reading students code - I do the same and it can be a great pain. But if you do it, you can give individual feedback and you'll understand, which are the most common mistatkes and adress them in your course.
  
  One compromise could be to offer different test sets, let's say some tests for each function needed for the complete task. That would allow to give points partially, but still you can not give points for a correct recursion, loop or whatever.
  
  An additional thought: You might consider giving a few test cases so the students can at least test if their code understands the input and delivers the output in a format your test understands. I don't know if this is a real world problem with your students, but it would be an issue with mine ;-).
  
  
  ---
  
  ### Answer by username_7 on 2019-05-14. Score: 2
  
  The 'external view' is important as it ensures the submission is correct. These tests should be similar to Acceptance Testing since you are checking whether the submission meets the 'business' requirements i.e. the requirements laid out in your question. I highly recommend automating this process with unit tests or console based tests (check output matches the expected output).
  
  The 'internal view' is what motivates your students to do better since only a person can judge this properly. Students are not motivated to write good code when they are only being judged by automated tests. I recommend looking over each student's submission and ensuring that they have followed the correct approach have met proper code quality standards.
  
  Of course, you can't give a lot of time to looking over each submission UNLESS the 'external view' tests are automated and you don't have to download and run the code for the 'internal view' reviews.
  
  I suggest automating your process using [AutoGradr](https://autogradr.com) (disclaimer: I am the author of this free tool)
  
  
  ---
  
  ### Answer by username_4 on 2018-12-12. Score: -2
  
  Unless the objective of the exercise is to copy an implementation (say from Java to C++), the "similarity" to any other implementation should have no weight in the grading. At least not in a positive sense. The similarity to another implementation could, however, be a warning sign for plagiarism. Everything else in the question is *up for debate*, so to speak.
  
  The "fairness" of the grading only hangs on two principles: Consistent application of standards and advance notice of those standards. If the same standard, or evaluation, is applied to all the work received, and the assigned grade would be the same regardless of which student turned it in, or which TA assigned the grade, then the standards are applied consistently. Prior to the examination, or assignment, the students need to be informed as to how their work will be evaluated. In the course syllabus, or in the beginning of the course is the best time to inform them, of course. If they have been given such a notice, and then their work is evaluated according to that guide, there should be nothing to object to - from a "fairness" point of view. Knowing that there will be unit tests, even without knowing what those might be, is still acceptable, so long as the same unit tests are applied in a uniform manner to all the work evaluated.
  
  How "fair" the evaluations are, however, likely has very small significance to how well the evaluations move the students toward the course objectives. Likewise, how such projects would be evaluated in a real-world workplace environment likely does not move them in the direction of meeting the course objectives. Of course, if the class is the **final** course in a series, the objective of which is to produce work-ready employees, then a purely external view of *works* vs. *fails* might be appropriate. In all other cases, however, such an extreme approach is not a very pedagogical approach. (As a side note, a binary go/no-go grade does not fit well into a scaled grading system either.)
  
  In an idealized world, the student projects are a chance for the instructor to determine how well the student has apprehended the course materials thus far, discover where the student has stumbled, and take corrective measures to return the student to where they should be relative to the course materials. Teachers, of every stripe, are an over-worked, under-paid, unappreciated lot. (The TA's get to share in that "glory", and will become just as over-worked as the instructors.) As such many of the opportunities to learn about the difficulties and misunderstandings the students have will be missed when grading assignments. Having a rubric, published by preference, which the TA can apply to the evaluation often helps.
  
  The course objectives, naturally, have greater weight in the rubric than other criteria. Yet non-course objectives should be included. While seldom included in the course specifications, clean code, naming conventions, documentation or self-documenting code, and adherence to coding style can all be covered by the rubric, and encouraged, thereby, in the students' work. Requiring adherence to such guidelines benefits the instructor (and TA's) in applying fine-grained grading evaluations. It can also aid the students in completing the projects, obtaining better results from the code, ease the burden of debugging the code prior to submission, and developing good habits for when they do enter the workforce, where go/no-go evaluations really matter. (I prefer to give more points to a clean almost-working program, than to a impenetrable kludge that passes every known test. The former can be maintained, including fixing the current version. The latter can probably only be maintained after being re-coded in some understandable manner.)
  
  Unfortunately, my view is only one way of seeing the job of "instructor." As to *it is okay to grade programming assignments based on this external view*, the answer is; "While it is less than ideal, yes, it is okay to do so."
  
  
  ---
  
  