Answers.Body: <p>You can standardize the testing library, build and test commands, and then use a &quot;meta&quot; test suite to run the student's test suite as a subprocess, asserting on the stdout/stderr that a certain number of tests exist and passed. You'll probably want to provide a sample meta-test suite that the student can run, or some submission judge that lets the student ensure their stdout is properly formatted and detected by the meta test suite.</p>
  <p>The problem here is that there's no guarantee of coverage or exercising interesting functionality. The student can fool your &quot;naive&quot; meta-test suite with a bunch of <code>assertEquals(1, 1)</code> statements. You can start by using a coverage tool to make sure the student's tests are actually running the app.</p>
  <p>To ensure false positives aren't passing, you could ask the student to include a standardized config file for their test suite that provides a set of &quot;bad&quot; test fixtures that will cause the same suite that passed before to fail. This ensures that the tests actually test what they claim to (passing on correct input, failing on incorrect input).</p>
  <p>Even with all this, students may still write a totally trivial app class to be tested, basically a silly &quot;return whatever makes the tests pass or fail&quot; app. A test that the coverage report shows a minimum number of lines executed can be circumvented. We haven't yet enforced a certain level of application complexity or purpose. That could be done by hand or by LLM, but LLM verification has obvious downsides of false positives and negatives and would need to be developed and spot checked very carefully.</p>
  <p>Non-LLM tools (like Sonar) exist to check the complexity of software (we don't really want to promote high cyclomatic complexity, per se, but we do want a certain amount of application complexity). You can also scale back some of the design freedom to help ensure the app has a basic level of complexity (tested with a traditional, direct unit test on the student's app, running alongside your meta-test which tests the student's test suite).</p>
  <p>For example, you might require that the project is a TODO list class where you can add/remove tasks, then require that it fulfills a basic level of functionality passing a standard unit test. Beyond that, give the student creative freedom to add their own specification, functionality and custom tests, which will be evaluated by the meta-test suite. Granted, this is a good deal more rigid than a pure greenfield project, but there may be some creative balance to be struck here (maybe giving them 4-5 different core project types to build on), and it can save you a lot of work on developing reliable meta tests.</p>
  <p>Needless to say, setting all of this up is a fair amount of work and won't be flawless, but automatically verifying that tests really work while allowing the student to design as they please is not a trivial task. I'd try to automate whatever is feasible to automate, then manually check the rest. Hopefully the partial automation will save enough of the manual work to make it worthwhile.</p>
  <hr />
  <p>If writing a meta-test suite is out of scope, try to formalize the project environment, structure, libraries, testing commands and output formats as much as possible. As above, ask students to provide standardized passing and failing fixtures and use required build and run commands. You could provide this code setup as a very loose skeleton. This gives students plenty of design flexibility, but (hopefully) provides enough consistency for graders to quickly run and evaluate each submission.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-11T16:59:32.863Z
Answers.LastActivityDate: 2024-06-13T04:45:23.423Z
Answers.LastEditDate: 2024-06-13T04:45:23.423Z
Answers.UserDisplayName: ggorlen
Answers.Body: <p>In my software engineering course, I created a series of worksheets that begin with practicing various forms of test analysis such as ECP and BVA.  We go over these worksheets in class together in depth.</p>
  <p>I then provide classes to test without any particular analysis (use all of the tools at your disposal, plus your logical facilities, to create a good testing suite for this class.)</p>
  <p>After that, they create a small, free-choice project on their own that I give them several weeks for.  Along the way, they must choose two classes from their project that they will analyze and create robust test suites (without becoming brittle).  They do a writeup of their analysis and justify their choices.  I grade them on the project cursorily, but in much more depth on this writeup.</p>
  <p>As part of the class, students create large group projects. I never have them create an entire test suite for an entire project because it is, frankly, too daunting. But everyone is responsible to choose one class from the project to create unit tests, and one system to create integration tests.  These are then graded.</p>
  <p>However, I believe that my approach lends itself well to the classroom.  The projects are of manageable size, the kids certainly learn about testing, the kids get a great deal of say in how they approach their work, and it's fairly easy to create assessment questions.  (Harder to grade: perform an ECP analysis on this method, and create a list of tests that ECP would suggest.  Easier to grade: which method(s) of analysis would suggest the following test for the method below?)</p>
  <p>I do not particularly teach or emphasize specification within my course, but a similar approach could certainly be taken.  Study the aspects of specification, do up a series of specifications, and have them pre-plan and pre-write specifications for a single system within their project.  Again, I would not make them specify every part - that is too much for a project done in school over just a few weeks.  But pre-specifying a single system within the larger project is certainly doable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-11T02:03:30.270Z
Answers.LastActivityDate: 2024-06-18T01:15:08.840Z
Answers.LastEditDate: 2024-06-18T01:15:08.840Z
Answers.UserDisplayName: Ben I.
Body: <p>At our university, we teach a course on OOP in Java. Currently, our project focuses on unit testing and specifications. However, to achieve this without too much manual effort, the students did not have a lot of freedom with regard to design.</p>
  <p>Would there be project structures that allow to focus on unit testing and design and can be corrected without too much manual effort, but don't need the rigid structure that comes with defining the whole class structure beforehand?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-10T21:41:14.127Z
Comments.Text: What other agile practices do you encourage/require beyond unit testing?
Comments.UserDisplayName: Buffy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-11T16:07:50.773Z
Comments.Text: Agile is not a goal of this course. We want to encourage thinking about modularity and how programs behave at the relevant interfaces (what leaks accross them).
Comments.UserDisplayName: Emiel Lanckriet
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-11T20:12:22.200Z
Comments.Text: So, is unit testing done after the code is written or as a means of getting it right first (i.e. Test Driven Development)?
Comments.UserDisplayName: Buffy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-12T15:19:10.720Z
Comments.Text: I'm having trouble understanding your process and so can't offer suggestions. In particular, if you aren't using agile practices, what is the "unit" exactly that is the basis of "unit testing"? Also, and just as vital, do students get a complete "specification" of the desired program at the beginning or is it done piece by piece (aka "stories")?
Comments.UserDisplayName: Buffy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-12T17:12:07.230Z
Comments.Text: The unit testing is done afterwards and where unit just means test they should try to test as little as possible per test. They are currently also expected to write formal specification based on the assignment in text form.
Comments.UserDisplayName: Emiel Lanckriet
Tags: <object-oriented-paradigm><unit-testing>
Title: Automating grading of projects with OOP and unit tests, but without imposing rigid design requirements
ThreadText: # Automating grading of projects with OOP and unit tests, but without imposing rigid design requirements
  
  (Asked by: username_0 on 2024-06-10)
  
  At our university, we teach a course on OOP in Java. Currently, our project focuses on unit testing and specifications. However, to achieve this without too much manual effort, the students did not have a lot of freedom with regard to design.
  
  Would there be project structures that allow to focus on unit testing and design and can be corrected without too much manual effort, but don't need the rigid structure that comes with defining the whole class structure beforehand?
  
  **Tags:** object-oriented-paradigm, unit-testing
  
  > **Comments:**
  > **username_1**: What other agile practices do you encourage/require beyond unit testing?  
  > **username_0**: Agile is not a goal of this course. We want to encourage thinking about modularity and how programs behave at the relevant interfaces (what leaks accross them).  
  > **username_1**: So, is unit testing done after the code is written or as a means of getting it right first (i.e. Test Driven Development)?  
  > **username_1**: I'm having trouble understanding your process and so can't offer suggestions. In particular, if you aren't using agile practices, what is the "unit" exactly that is the basis of "unit testing"? Also, and just as vital, do students get a complete "specification" of the desired program at the beginning or is it done piece by piece (aka "stories")?  
  > **username_0**: The unit testing is done afterwards and where unit just means test they should try to test as little as possible per test. They are currently also expected to write formal specification based on the assignment in text form.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-06-11. Score: 2
  
  You can standardize the testing library, build and test commands, and then use a "meta" test suite to run the student's test suite as a subprocess, asserting on the stdout/stderr that a certain number of tests exist and passed. You'll probably want to provide a sample meta-test suite that the student can run, or some submission judge that lets the student ensure their stdout is properly formatted and detected by the meta test suite.
  
  The problem here is that there's no guarantee of coverage or exercising interesting functionality. The student can fool your "naive" meta-test suite with a bunch of `assertEquals(1, 1)` statements. You can start by using a coverage tool to make sure the student's tests are actually running the app.
  
  To ensure false positives aren't passing, you could ask the student to include a standardized config file for their test suite that provides a set of "bad" test fixtures that will cause the same suite that passed before to fail. This ensures that the tests actually test what they claim to (passing on correct input, failing on incorrect input).
  
  Even with all this, students may still write a totally trivial app class to be tested, basically a silly "return whatever makes the tests pass or fail" app. A test that the coverage report shows a minimum number of lines executed can be circumvented. We haven't yet enforced a certain level of application complexity or purpose. That could be done by hand or by LLM, but LLM verification has obvious downsides of false positives and negatives and would need to be developed and spot checked very carefully.
  
  Non-LLM tools (like Sonar) exist to check the complexity of software (we don't really want to promote high cyclomatic complexity, per se, but we do want a certain amount of application complexity). You can also scale back some of the design freedom to help ensure the app has a basic level of complexity (tested with a traditional, direct unit test on the student's app, running alongside your meta-test which tests the student's test suite).
  
  For example, you might require that the project is a TODO list class where you can add/remove tasks, then require that it fulfills a basic level of functionality passing a standard unit test. Beyond that, give the student creative freedom to add their own specification, functionality and custom tests, which will be evaluated by the meta-test suite. Granted, this is a good deal more rigid than a pure greenfield project, but there may be some creative balance to be struck here (maybe giving them 4-5 different core project types to build on), and it can save you a lot of work on developing reliable meta tests.
  
  Needless to say, setting all of this up is a fair amount of work and won't be flawless, but automatically verifying that tests really work while allowing the student to design as they please is not a trivial task. I'd try to automate whatever is feasible to automate, then manually check the rest. Hopefully the partial automation will save enough of the manual work to make it worthwhile.
  
  ---
  
  If writing a meta-test suite is out of scope, try to formalize the project environment, structure, libraries, testing commands and output formats as much as possible. As above, ask students to provide standardized passing and failing fixtures and use required build and run commands. You could provide this code setup as a very loose skeleton. This gives students plenty of design flexibility, but (hopefully) provides enough consistency for graders to quickly run and evaluate each submission.
  
  
  ---
  
  ### Answer by username_3 on 2024-06-11. Score: 1
  
  In my software engineering course, I created a series of worksheets that begin with practicing various forms of test analysis such as ECP and BVA. We go over these worksheets in class together in depth.
  
  I then provide classes to test without any particular analysis (use all of the tools at your disposal, plus your logical facilities, to create a good testing suite for this class.)
  
  After that, they create a small, free-choice project on their own that I give them several weeks for. Along the way, they must choose two classes from their project that they will analyze and create robust test suites (without becoming brittle). They do a writeup of their analysis and justify their choices. I grade them on the project cursorily, but in much more depth on this writeup.
  
  As part of the class, students create large group projects. I never have them create an entire test suite for an entire project because it is, frankly, too daunting. But everyone is responsible to choose one class from the project to create unit tests, and one system to create integration tests. These are then graded.
  
  However, I believe that my approach lends itself well to the classroom. The projects are of manageable size, the kids certainly learn about testing, the kids get a great deal of say in how they approach their work, and it's fairly easy to create assessment questions. (Harder to grade: perform an ECP analysis on this method, and create a list of tests that ECP would suggest. Easier to grade: which method(s) of analysis would suggest the following test for the method below?)
  
  I do not particularly teach or emphasize specification within my course, but a similar approach could certainly be taken. Study the aspects of specification, do up a series of specifications, and have them pre-plan and pre-write specifications for a single system within their project. Again, I would not make them specify every part - that is too much for a project done in school over just a few weeks. But pre-specifying a single system within the larger project is certainly doable.
  
  
  ---
  
  