Answers.Body: <p>First, let me congratulate you on a really thorough grading procedure for such a large class.  I am impressed!  I think that you have created an excellent balancing act among the realities of grading code among a large population, and I'm sure that your students benefit from it tremendously.</p>
  <p>Now, you wrote:</p>
  <blockquote>
  <p>about half the students wrote perfect code (100% in all rubrics), and they all used very similar ideas - similar tricks, algorithms, etc.</p>
  </blockquote>
  <p>It reminds me very much of this:</p>
  <blockquote>
  <p>All happy families are alike, but every unhappy family is unhappy in its own way</p>
  <p>―Leo Tolstoy, Anna Karenina, 1878</p>
  </blockquote>
  <p>And I think it's true.  In any happy family, partners agree upon roles together.  Partners largely respect and listen to one another.  Children are taught how to be in the world in a manner forged from love.  The path to familial harmony is relatively narrow compared to the infinity of ways that things can go terribly wrong.</p>
  <p>Turning back to code, while it's true that there are an infinity of possible answers to almost any coding prompt, it is also true that there are <em>very strong norms</em> to good, clean code.  Many of your students will absorb these norms, so it makes sense, even without any collusion, that you will see many highly similar answers among those who do well.</p>
  <p>In addition, algorithms are far more proscribed than the contents of paragraphs.  In the end, they must work, and they should be understandable to others as well.</p>
  <p>Once you take into account readability, there are only so many reasonable approaches to finding the maximum element in an array, memoizing a function, or performing a DFS on a tree.</p>
  <p>So I guess I'd suggest that, at least from the evidence you wrote into the question, nothing is amiss, and you needn't worry.  Instead, keep up your excellent grading procedure!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-11T22:35:19.740Z
Answers.Text: As an aside, I remember hearing a social scientist explaining that Tolstoy was wrong about this: that unhappy families all tend to have the same few problems, while happy families all find different ways of working them out. (Your answer still stands, despite that, of course!)
Answers.UserDisplayName: gidds
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T05:16:27.187Z
Answers.Text: @gidds The quote is certainly glib, but I think that, in truth, everyone and every situation is unique.  You have to apply a grouping lens to see commonality, but truly, that's an artifact of our own brains.  We *really* like to group things.  In truth, I am quite sure there are exactly as many ways people are unhappy as there are unhappy people, and the same can be said for ways to be happy as well.  Tolstoy and your professor were applying different lenses to map their understanding, but the map is not the territory.
Answers.UserDisplayName: Ben I.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T14:32:43.657Z
Answers.Text: Thanks for the feedback. I understand that there is a limited number of perfect solutions. But I still feel that the current grading scheme does not do much to encourage originality and self-effort. In fact, it encourages students to just ask the top students in class to share their ideas with the group. I am trying to tweak the scheme a bit.
Answers.UserDisplayName: Erel Segal-Halevi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-11T18:55:11.800Z
Answers.LastActivityDate: 2022-01-11T18:55:11.800Z
Answers.UserDisplayName: Ben I.
Answers.Body: <p>Let me suggest two things, but I don't know if they will solve the problem.</p>
  <p>The first is that once you set the rubric you are stuck with it until you publicly replace it. You may be right that there was collaboration but your &quot;feelings&quot; can't be acted on without evidence. I think you understand that. If it only mentions quality and completeness then it is that which you must depend on. But, you could also require explanations of things as part of any assignment. In Java, we have JavaDocs for that where we describe the intent of any class and of important methods. They are harder to grade, but also harder to copy in any real way, since they use natural language.</p>
  <p>A short exposition, say a page, of what an assignment has accomplished and how could be submitted along with each assignment. Ideally it could include the &quot;why&quot; of major decisions, such as algorithm choice. Students would be required to explain their work, not just provide it.</p>
  <p>The second thing is to spell out explicitly (for the course, and perhaps for individual assignments) what sorts of collaboration are permitted and which are not. But to make this effective (hmmm, <em>more</em> effective) you need to get them to actually affirm that they will respect it in all details, both the letter and the intent. It is one thing to say to students &quot;You must do this.&quot; and quite a different thing if the student says to you &quot;I agree that I will do this.&quot;</p>
  <p>But in any conversation (oral or written) about an honor code, you need to actually explain why the rules are in place. After all, in the real world, collaboration is expected and the quality of the product is paramount. In an educational environment, it is the learning that is paramount, not the product. Students don't always understand that you didn't ask them to do something because you want the result. After all, you could have produced it yourself much more easily. Emphasize that any restrictions are to help ensure their skill development.</p>
  <p>And, of course, if you forbid them from communicating with one another they you almost certainly need to provide ways for them to communicate with you. My technique for this was a private mailing list that all staff and students were subscribed to. Students could ask questions (but not post solutions/code) to the list. The questions would get answered, perhaps by another student - in public. Everyone saw every question and every answer. This has the additional educational benefit that when one student asks a question it is likely that others do also, but hesitated to ask for whatever reason. No one needed to feel &quot;stuck&quot; and then tempted to break the rules.</p>
  <p>Like many things, students need to be taught how to behave.</p>
  <hr />
  <p>Note, of course, that the above doesn't address &quot;originality&quot; explicitly. If you want to encourage that, you may want to find a way to get that idea into your rubric. But you would need to be careful to define what you mean. Variable names taken from Pokemon probably isn't helpful. One way to make this work is to offer a few points in the rubric for &quot;extensions&quot; to the assignment. They would have to be explained in natural language at least, and, perhaps programmed. But a few points for suggesting some &quot;novel&quot; extension to the assignment might work and be hard to fake if arrived at collaboratively.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T10:59:02.760Z
Answers.Text: I do not want to forbid collaboration altogether. I just want to reward students who did a larger effort to solve the exercise on their own. The idea of "extensions" seems very useful. I can design an exercise with 80% "fixed part" (e.g. write a binary tree class) and 20% "variable part" (e.g. write a function that prints out the tree in a nice format). Since 'nice' format is subjective, I can expect to have more originality (of course, the variable part cannot be graded automatically, only during the interview).
Answers.UserDisplayName: Erel Segal-Halevi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-12T12:02:36.173Z
Answers.Text: "*After all, in the real world, collaboration is expected and the quality of the product is paramount. In an educational environment, it is the learning that is paramount, not the product. Students don't always understand that you didn't ask them to do something because you want the result. After all, you could have produced it yourself much more easily.*" - I **really** wish StackOverflow would understand that. There are quite a few good quality questions which explain that OP can't use feature `X` and SO generally goes crazy about how dumb teachers are for forbidding using particular tools.
Answers.UserDisplayName: Fureeish
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-11T20:08:47.233Z
Answers.LastActivityDate: 2022-01-11T20:55:20.553Z
Answers.LastEditDate: 2022-01-11T20:55:20.553Z
Answers.UserDisplayName: Buffy
Body: <p>BACKGROUND: I teach a C++ course with 300 students where the grade is based on programming assignments. We use automatic tools to assess the assignment correctness (using many unit-tests), code quality (using <code>clang-tidy</code>), and to check plagiarism. In addition, the students present some of their solutions to the lecturer or TA in an online interview, and are graded by six quality criteria: organization, testing, correctness, efficiency, readability, and safety (I used recommendations from <a href="https://academia.stackexchange.com/">here</a> for the rubric).</p>
  <p>Last year, while I was interviewing students, I noticed a paradoxical phoenomenon: about half the students wrote perfect code (100% in all rubrics), and they all used very similar ideas - similar tricks, algorithms, etc. While the solutions were not entirely identical (so I cannot blame them for plagiarism), I was quite convinced that some of them shared their ideas with others. On the other hand, about half the students wrote mediocre code (60-70% in most rubrics), and their work was very different than others - I was quite convinced that they did all the work by themselves.</p>
  <p>It turned out that I gave a lower grade to students who did all work by themselves, than to students who (probably) consulted other students and used their ideas. My grading scheme does not properly encourage students to do original work. Note that the course guidelines do not forbid students to consult each other. But, I still want to encourage students to make the best effort to solve the homework by themselves.</p>
  <p>QUESTION: how can I design a grading scheme for programming exercises, which simultaneously encourage both quality and originality?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-11T16:30:49.467Z
Comments.Text: Give a bit more background. What is the student/staff ratio for such a large course? How many TAs/graders etc. are involved? I think that other questions you've ask suggest you have a near impossible task.
Comments.UserDisplayName: Buffy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-11T17:34:09.960Z
Comments.Text: @Buffy students present their homework in "lab" hours. There are 15 lab hours per week, so 20 students per lab. Assuming 5 presentations per hour, each student gets to present once in 4 weeks, which is 3 times per term (so they can present 3 out of 5 assignments). Indeed, it is very challenging. I asked to have 13 students per lab, but it was not approved due to budget considerations.
Comments.UserDisplayName: Erel Segal-Halevi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-11T17:38:40.010Z
Comments.Text: So, are you alone managing this class?
Comments.UserDisplayName: Buffy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-11T18:09:57.233Z
Comments.Text: @Buffy I am currently the lecturer in charge. There are two other lecturers that are responsible for teaching, and 5 TAs that are responsible for tutoring and handling the labs.
Comments.UserDisplayName: Erel Segal-Halevi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-11T18:26:25.987Z
Comments.Text: I have a strong sympathy with this question -- I've on occasion disliked myself for the grades I gave
Comments.UserDisplayName: Rushi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-11T23:35:39.973Z
Comments.Text: Is there a problem? You used the term "mediocre code" for the students that turned out to have worked alone. I'm fine with students consulting each other. In fact, I tell them to formalize grouping of up to 3 and submit just once, with all their names on it. I'm sort of assuming that they themselves will deal with the ones not pulling their weight.
Comments.UserDisplayName: Victor Eijkhout
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-12T10:35:11.630Z
Comments.Text: @VictorEijkhout Until two years ago I, too, encouraged group submissions. Then, due to COVID we had to do oral examinations. I found out that some students, who got 100% homework grade, could not write a single line of code. When I asked "how did you get 100% homework without writing code?" they said  "my partner wrote all the code. My job was just to submit the solution to Moodle"... Since then, I switched to individual submission.
Comments.UserDisplayName: Erel Segal-Halevi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-12T10:51:43.537Z
Comments.Text: @GuyCoder students get 1 point for the automatic submission, and 16 points for the interview, where they also get personal feedback on their submission. It is important for students to know exactly how their grade is computed - this is why we need a transparent grading scheme.
Comments.UserDisplayName: Erel Segal-Halevi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-17T00:08:45.387Z
Comments.Text: Make 30% of the grade be for originality? It is an old issue that the ones getting As are basically just good at regurgitating and gaming the school, and the capable students are annoyed, withdrawn, misunderstood and told they will never amount to anything, like that guy... Uh... Einstein, that's it. How to solve this? Switch to apprenticeship. Ramanujan had an individual tutor who recognized his brilliance, or else he might have spent his life in the village where he was born.
Comments.UserDisplayName: Scott Rowe
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-17T09:25:37.393Z
Comments.Text: @ScottRowe We have a kind of apprenticeship at the 3rd year, when students do individual projects. The course I am talking about is 2nd year, it should give students basic tools for doing projects.
Comments.UserDisplayName: Erel Segal-Halevi
Tags: <grading><homework>
Title: Grading programming exercises: the quality vs. originality paradox
ThreadText: # Grading programming exercises: the quality vs. originality paradox
  
  (Asked by: username_0 on 2022-01-11)
  
  BACKGROUND: I teach a C++ course with 300 students where the grade is based on programming assignments. We use automatic tools to assess the assignment correctness (using many unit-tests), code quality (using `clang-tidy`), and to check plagiarism. In addition, the students present some of their solutions to the lecturer or TA in an online interview, and are graded by six quality criteria: organization, testing, correctness, efficiency, readability, and safety (I used recommendations from [here](https://academia.stackexchange.com/) for the rubric).
  
  Last year, while I was interviewing students, I noticed a paradoxical phoenomenon: about half the students wrote perfect code (100% in all rubrics), and they all used very similar ideas - similar tricks, algorithms, etc. While the solutions were not entirely identical (so I cannot blame them for plagiarism), I was quite convinced that some of them shared their ideas with others. On the other hand, about half the students wrote mediocre code (60-70% in most rubrics), and their work was very different than others - I was quite convinced that they did all the work by themselves.
  
  It turned out that I gave a lower grade to students who did all work by themselves, than to students who (probably) consulted other students and used their ideas. My grading scheme does not properly encourage students to do original work. Note that the course guidelines do not forbid students to consult each other. But, I still want to encourage students to make the best effort to solve the homework by themselves.
  
  QUESTION: how can I design a grading scheme for programming exercises, which simultaneously encourage both quality and originality?
  
  **Tags:** grading, homework
  
  > **Comments:**
  > **username_1**: Give a bit more background. What is the student/staff ratio for such a large course? How many TAs/graders etc. are involved? I think that other questions you've ask suggest you have a near impossible task.  
  > **username_0**: @username_1 students present their homework in "lab" hours. There are 15 lab hours per week, so 20 students per lab. Assuming 5 presentations per hour, each student gets to present once in 4 weeks, which is 3 times per term (so they can present 3 out of 5 assignments). Indeed, it is very challenging. I asked to have 13 students per lab, but it was not approved due to budget considerations.  
  > **username_1**: So, are you alone managing this class?  
  > **username_0**: @username_1 I am currently the lecturer in charge. There are two other lecturers that are responsible for teaching, and 5 TAs that are responsible for tutoring and handling the labs.  
  > **username_2**: I have a strong sympathy with this question -- I've on occasion disliked myself for the grades I gave  
  > **username_3**: Is there a problem? You used the term "mediocre code" for the students that turned out to have worked alone. I'm fine with students consulting each other. In fact, I tell them to formalize grouping of up to 3 and submit just once, with all their names on it. I'm sort of assuming that they themselves will deal with the ones not pulling their weight.  
  > **username_0**: @username_3 Until two years ago I, too, encouraged group submissions. Then, due to COVID we had to do oral examinations. I found out that some students, who got 100% homework grade, could not write a single line of code. When I asked "how did you get 100% homework without writing code?" they said  "my partner wrote all the code. My job was just to submit the solution to Moodle"... Since then, I switched to individual submission.  
  > **username_0**: @GuyCoder students get 1 point for the automatic submission, and 16 points for the interview, where they also get personal feedback on their submission. It is important for students to know exactly how their grade is computed - this is why we need a transparent grading scheme.  
  > **username_4**: Make 30% of the grade be for originality? It is an old issue that the ones getting As are basically just good at regurgitating and gaming the school, and the capable students are annoyed, withdrawn, misunderstood and told they will never amount to anything, like that guy... Uh... Einstein, that's it. How to solve this? Switch to apprenticeship. Ramanujan had an individual tutor who recognized his brilliance, or else he might have spent his life in the village where he was born.  
  > **username_0**: @username_4 We have a kind of apprenticeship at the 3rd year, when students do individual projects. The course I am talking about is 2nd year, it should give students basic tools for doing projects.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-01-11. Score: 4 | Accepted answer
  
  Let me suggest two things, but I don't know if they will solve the problem.
  
  The first is that once you set the rubric you are stuck with it until you publicly replace it. You may be right that there was collaboration but your "feelings" can't be acted on without evidence. I think you understand that. If it only mentions quality and completeness then it is that which you must depend on. But, you could also require explanations of things as part of any assignment. In Java, we have JavaDocs for that where we describe the intent of any class and of important methods. They are harder to grade, but also harder to copy in any real way, since they use natural language.
  
  A short exposition, say a page, of what an assignment has accomplished and how could be submitted along with each assignment. Ideally it could include the "why" of major decisions, such as algorithm choice. Students would be required to explain their work, not just provide it.
  
  The second thing is to spell out explicitly (for the course, and perhaps for individual assignments) what sorts of collaboration are permitted and which are not. But to make this effective (hmmm, *more* effective) you need to get them to actually affirm that they will respect it in all details, both the letter and the intent. It is one thing to say to students "You must do this." and quite a different thing if the student says to you "I agree that I will do this."
  
  But in any conversation (oral or written) about an honor code, you need to actually explain why the rules are in place. After all, in the real world, collaboration is expected and the quality of the product is paramount. In an educational environment, it is the learning that is paramount, not the product. Students don't always understand that you didn't ask them to do something because you want the result. After all, you could have produced it yourself much more easily. Emphasize that any restrictions are to help ensure their skill development.
  
  And, of course, if you forbid them from communicating with one another they you almost certainly need to provide ways for them to communicate with you. My technique for this was a private mailing list that all staff and students were subscribed to. Students could ask questions (but not post solutions/code) to the list. The questions would get answered, perhaps by another student - in public. Everyone saw every question and every answer. This has the additional educational benefit that when one student asks a question it is likely that others do also, but hesitated to ask for whatever reason. No one needed to feel "stuck" and then tempted to break the rules.
  
  Like many things, students need to be taught how to behave.
  
  ---
  
  Note, of course, that the above doesn't address "originality" explicitly. If you want to encourage that, you may want to find a way to get that idea into your rubric. But you would need to be careful to define what you mean. Variable names taken from Pokemon probably isn't helpful. One way to make this work is to offer a few points in the rubric for "extensions" to the assignment. They would have to be explained in natural language at least, and, perhaps programmed. But a few points for suggesting some "novel" extension to the assignment might work and be hard to fake if arrived at collaboratively.
  
  > **Comments:**
  > **username_0**: I do not want to forbid collaboration altogether. I just want to reward students who did a larger effort to solve the exercise on their own. The idea of "extensions" seems very useful. I can design an exercise with 80% "fixed part" (e.g. write a binary tree class) and 20% "variable part" (e.g. write a function that prints out the tree in a nice format). Since 'nice' format is subjective, I can expect to have more originality (of course, the variable part cannot be graded automatically, only during the interview).  
  > **username_7**: "*After all, in the real world, collaboration is expected and the quality of the product is paramount. In an educational environment, it is the learning that is paramount, not the product. Students don't always understand that you didn't ask them to do something because you want the result. After all, you could have produced it yourself much more easily.*" - I **really** wish StackOverflow would understand that. There are quite a few good quality questions which explain that OP can't use feature `X` and SO generally goes crazy about how dumb teachers are for forbidding using particular tools.  
  
  ---
  
  ### Answer by username_5 on 2022-01-11. Score: 3
  
  First, let me congratulate you on a really thorough grading procedure for such a large class. I am impressed! I think that you have created an excellent balancing act among the realities of grading code among a large population, and I'm sure that your students benefit from it tremendously.
  
  Now, you wrote:
  
  > about half the students wrote perfect code (100% in all rubrics), and they all used very similar ideas - similar tricks, algorithms, etc.
  
  It reminds me very much of this:
  
  > All happy families are alike, but every unhappy family is unhappy in its own way
  >
  > ―Leo Tolstoy, Anna Karenina, 1878
  
  And I think it's true. In any happy family, partners agree upon roles together. Partners largely respect and listen to one another. Children are taught how to be in the world in a manner forged from love. The path to familial harmony is relatively narrow compared to the infinity of ways that things can go terribly wrong.
  
  Turning back to code, while it's true that there are an infinity of possible answers to almost any coding prompt, it is also true that there are *very strong norms* to good, clean code. Many of your students will absorb these norms, so it makes sense, even without any collusion, that you will see many highly similar answers among those who do well.
  
  In addition, algorithms are far more proscribed than the contents of paragraphs. In the end, they must work, and they should be understandable to others as well.
  
  Once you take into account readability, there are only so many reasonable approaches to finding the maximum element in an array, memoizing a function, or performing a DFS on a tree.
  
  So I guess I'd suggest that, at least from the evidence you wrote into the question, nothing is amiss, and you needn't worry. Instead, keep up your excellent grading procedure!
  
  > **Comments:**
  > **username_6**: As an aside, I remember hearing a social scientist explaining that Tolstoy was wrong about this: that unhappy families all tend to have the same few problems, while happy families all find different ways of working them out. (Your answer still stands, despite that, of course!)  
  > **username_5**: @username_6 The quote is certainly glib, but I think that, in truth, everyone and every situation is unique.  You have to apply a grouping lens to see commonality, but truly, that's an artifact of our own brains.  We *really* like to group things.  In truth, I am quite sure there are exactly as many ways people are unhappy as there are unhappy people, and the same can be said for ways to be happy as well.  Tolstoy and your professor were applying different lenses to map their understanding, but the map is not the territory.  
  > **username_0**: Thanks for the feedback. I understand that there is a limited number of perfect solutions. But I still feel that the current grading scheme does not do much to encourage originality and self-effort. In fact, it encourages students to just ask the top students in class to share their ideas with the group. I am trying to tweak the scheme a bit.  
  
  ---
  
  