Answers.Body: <p>Very interesting strategy. You are essentially putting the 10% test case-writers in the role of a sophisticated recruiting agent who is "interviewing" the rest of the 90% of the class for a job. It is probably not really fair, since the 10% are probably not sufficiently mature to identify the key concepts to be tested, but the class discussion (or perhaps outraged complaints?)  from the consequences (or should I say carnage?) should be extremely "rich". I hope you are ready for an interesting challenge.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-19T03:14:24.643Z
Answers.LastActivityDate: 2019-02-19T04:40:04.530Z
Answers.LastEditDate: 2019-02-19T04:40:04.530Z
Answers.UserDisplayName: 42-
Answers.Body: <p>If you run all implementations against all tests, then you can see anomalies. If you put results into a grid, then any failures should show up as a line: horizontal for implementations, vertical for tests. Any odd failures will still need investigating, but there should be few.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-16T17:22:12.540Z
Answers.Text: Interesting. Now that I think of it, I believe that there must be some research in the statistics community about such situation. Can be intereting to check.
Answers.UserDisplayName: Erel Segal-Halevi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-16T16:11:22.137Z
Answers.LastActivityDate: 2019-02-16T16:11:22.137Z
Answers.UserDisplayName: ctrl-alt-delor
Answers.Body: <p>Have you thought about having a solution that you create and test the students test cases against your solution? Then you have a stable platform for running the student defined tests.</p>
  
  <p>Also run test cases which you prepare against the student's code. Now you have graded the two parts independently of one another.</p>
  
  <p>Just out of curiosity, what did you use for auto-grading?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-16T16:04:31.360Z
Answers.Text: If you have created a working solution, then by definition you have a set of tests. (If you have not tested it, then any assertion that it is *working*, is just an opinion, and it is safe to ignore it.
Answers.UserDisplayName: ctrl-alt-delor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-16T17:23:52.030Z
Answers.Text: Regarding your question: I checked many platforms, but did not find anyone that allowed the students to submit an entire github repository as a solution. So I wrote one myself: https://github.com/erelsgl/badkan
Answers.UserDisplayName: Erel Segal-Halevi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-15T20:56:39.090Z
Answers.LastActivityDate: 2019-02-15T20:56:39.090Z
Answers.UserDisplayName: Fritz Sieker
Body: <p>Last year I taught a programming course, and used an automatic grading system for grading the students' homework (in addition to other evaluation measuares). It worked quite well; the bottleneck was the need to write test-cases for each assignment.</p>
  
  <p>This year, I would like to have the students themselves write the test-cases - both to save time, and to have them practice this important skill.</p>
  
  <p>I thought of the following scheme:</p>
  
  <ul>
  <li>In each of the 10 assignments, 10% of the students will be the "test set" and the other 90% will be the "train set".</li>
  <li>The "test set" students will write test-cases - each student will write e.g. 10 test-cases.</li>
  <li>The "train set" students will solve the assignment; their solutions will be tested using the test-cases of the test-set.</li>
  </ul>
  
  <p>The problem with this scheme is that, if the solution of student A from the train-set fails on the test-case of student B from the test-set, it is not clear whether it's because an error in student A's solution or an error in student B's test-case. Of course I can look and see who is correct, but this requires a lot of work and undermines the idea of automatic grading.</p>
  
  <p>How can I improve this scheme to allow automatic or almost-automatic grading of both homework assignments and test-cases?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-16T16:10:40.077Z
Comments.Text: I have been coaching a student, that has been given lab work with unit tests to pass. While I agree that this is a good idea, we found that we had to write our own tests. To do TDD: you have to keep asking what is the next test?; What test will lead to the simplest next step? I don't know if this can be predicted. If we make a different choice then the next choices start to diverge, event though we end up passing all the tests in the end.
Comments.UserDisplayName: ctrl-alt-delor
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-16T20:02:01.287Z
Comments.Text: Why not have all students write testcases and award a grade for each one, based on how the other students who answer it get on? If it has no errors then a full grade, less for errors. Have a set period of time for the writing activity, then move to students solving them - by then there should be so many that each student gets a random selection to answer.
Comments.UserDisplayName: Solar Mike
Tags: <grading><auto-graders><unit-testing><information-technology>
Title: Automatic grading of test-cases
ThreadText: # Automatic grading of test-cases
  
  (Asked by: username_0 on 2019-02-15)
  
  Last year I taught a programming course, and used an automatic grading system for grading the students' homework (in addition to other evaluation measuares). It worked quite well; the bottleneck was the need to write test-cases for each assignment.
  
  This year, I would like to have the students themselves write the test-cases - both to save time, and to have them practice this important skill.
  
  I thought of the following scheme:
  
  * In each of the 10 assignments, 10% of the students will be the "test set" and the other 90% will be the "train set".
  * The "test set" students will write test-cases - each student will write e.g. 10 test-cases.
  * The "train set" students will solve the assignment; their solutions will be tested using the test-cases of the test-set.
  
  The problem with this scheme is that, if the solution of student A from the train-set fails on the test-case of student B from the test-set, it is not clear whether it's because an error in student A's solution or an error in student B's test-case. Of course I can look and see who is correct, but this requires a lot of work and undermines the idea of automatic grading.
  
  How can I improve this scheme to allow automatic or almost-automatic grading of both homework assignments and test-cases?
  
  **Tags:** grading, auto-graders, unit-testing, information-technology
  
  > **Comments:**
  > **username_1**: I have been coaching a student, that has been given lab work with unit tests to pass. While I agree that this is a good idea, we found that we had to write our own tests. To do TDD: you have to keep asking what is the next test?; What test will lead to the simplest next step? I don't know if this can be predicted. If we make a different choice then the next choices start to diverge, event though we end up passing all the tests in the end.  
  > **username_2**: Why not have all students write testcases and award a grade for each one, based on how the other students who answer it get on? If it has no errors then a full grade, less for errors. Have a set period of time for the writing activity, then move to students solving them - by then there should be so many that each student gets a random selection to answer.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2019-02-15. Score: 4
  
  Have you thought about having a solution that you create and test the students test cases against your solution? Then you have a stable platform for running the student defined tests.
  
  Also run test cases which you prepare against the student's code. Now you have graded the two parts independently of one another.
  
  Just out of curiosity, what did you use for auto-grading?
  
  > **Comments:**
  > **username_1**: If you have created a working solution, then by definition you have a set of tests. (If you have not tested it, then any assertion that it is *working*, is just an opinion, and it is safe to ignore it.  
  > **username_0**: Regarding your question: I checked many platforms, but did not find anyone that allowed the students to submit an entire github repository as a solution. So I wrote one myself: https://github.com/erelsgl/badkan  
  
  ---
  
  ### Answer by username_1 on 2019-02-16. Score: 3
  
  If you run all implementations against all tests, then you can see anomalies. If you put results into a grid, then any failures should show up as a line: horizontal for implementations, vertical for tests. Any odd failures will still need investigating, but there should be few.
  
  > **Comments:**
  > **username_0**: Interesting. Now that I think of it, I believe that there must be some research in the statistics community about such situation. Can be intereting to check.  
  
  ---
  
  ### Answer by username_3 on 2019-02-19. Score: 2
  
  Very interesting strategy. You are essentially putting the 10% test case-writers in the role of a sophisticated recruiting agent who is "interviewing" the rest of the 90% of the class for a job. It is probably not really fair, since the 10% are probably not sufficiently mature to identify the key concepts to be tested, but the class discussion (or perhaps outraged complaints?) from the consequences (or should I say carnage?) should be extremely "rich". I hope you are ready for an interesting challenge.
  
  
  ---
  
  