Answers.Body: <p>Some of it depends on the level of the student. There's obviously a big difference between a freshman just starting out, a senior about to graduate, and someone who's been in industry for awhile.</p>
  
  <p>Keep in mind that, once your students are in industry, they're going to be expected to be able to write code that passes QA, and it's usually <em>not</em> the case that QA will supply them with a bunch of unit tests in advance. This being the case, it's essential that people learn to test their own stuff before they graduate.</p>
  
  <p>I'm of the opinion that students should be taught how to write proper unit tests as early as possible. This should eventually progress to other kinds of automated testing over time as appropriate, such as Selenium or Coded UI tests. It's also useful to be aware of major testing techniques and terminology, such as <a href="https://en.wikipedia.org/wiki/Boundary-value_analysis" rel="nofollow noreferrer">boundary value analysis</a>.</p>
  
  <p>With this said, it's probably helpful to give new students tests early on and then slowly "wean" them off of that.</p>
  
  <p>When I was in college, some of my instructors would also suggest test cases as part of the assignment.</p>
  
  <p>I also agree with @EllenSpertus's suggestions for having students submit test cases as part of the assignment.</p>
  
  <p>For learning purposes, you could start students out with "here's a piece of code, suggest 5 test cases for this" or "here's some code, write some unit tests for it." Again, you could start with giving them specific things to test for and progressively decreasing the amount you give them "up front" with the goal of eventually getting to "give me whatever automated tests you think are reasonable with your assignment" and grading them on how good and thorough the tests are.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T21:00:13.373Z
Answers.LastActivityDate: 2017-06-26T21:00:13.373Z
Answers.UserDisplayName: EJoshuaS - Stand with Ukraine
Answers.Body: <p>My best practice would be to provide students with test cases and require them to submit additional test cases with their code. Then run everyone's test cases against everyone's code. Let the students know that they will get extra credit if their tests break anyone's code. That will put them in a mindset of creating tough tests and writing bullet-proof code, rather than stopping when the pass the provided tests.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-25T18:26:20.693Z
Answers.Text: I really like this solution, seems like a great way to encourage students to think outside the box and write tests for situations that you may not have thought of initially.
Answers.UserDisplayName: Adam Williams
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T05:32:55.083Z
Answers.Text: But make sure the student-provided test cases *should* pass first.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T12:28:29.970Z
Answers.Text: A course at my Uni did this! There were two stages: At first we had to upload code that would be automatically tested against testcases and in the second stage we would upload testcases to run against the other students solutions. It was very fun to work against the testcases, but the second stage was to easy as most students had not beat many testcases from the first stage and also the objective wasn't clear (do we have to pass our own cases?). Also as @immibis said you should make sure that the test cases are correct by running them against a reference solution.
Answers.UserDisplayName: KarlKastor
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T17:47:09.980Z
Answers.Text: I feel like you would get too many badly written test cases that fail when they shouldn't.  At the least have what you consider to be a very well written sample version of the code that you run the test cases against.  I'd probably ignore every test case that fails a valid sample code.  Sure someone may have written a test case that actually caught a bug in your sample 'working' implementation, but most of the time they won't and it's not worth the effort of identifying which tests are valid and which aren't.
Answers.UserDisplayName: dsollen
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T21:51:42.177Z
Answers.Text: @KarlKastor Are bad tests flushed out by testing them against a canonical implementation of the program?
Answers.UserDisplayName: corsiKa
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T22:15:50.503Z
Answers.Text: @corsiKa In the Uni course I was talking about bad tests weren't 'flushed out', but you were limited to two tests and your implementation had to pass at least one of these. You were encouraged  to write cases that your own program would pass, so that your program would pass more cases. But I think it would have been better to check the cases against a reference implementation and flushing them out.
Answers.UserDisplayName: KarlKastor
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-27T00:44:08.760Z
Answers.Text: @KarlKastor Makes sense. It also is a good lesson in that you need to have good tests before you have good testing. Sometimes people forget that tests are not always right and not always good forever - sometimes you "break a test" and the functionality has actually changed. Quality is a goal, "making tests pass" never should be!
Answers.UserDisplayName: corsiKa
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-01T19:53:08.857Z
Answers.Text: When I have tried having students submit their own tests, they have been of very low quality in my experience. Often they are testing tiny corner cases which where there was some issue in the original spec.
  
  It's VERY hard to write specs where you cover every possible corner case, particular when it comes to things like what happens when integers overflow, or code runs out of memory.
Answers.UserDisplayName: Chris Jefferson
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-25T18:22:53.127Z
Answers.LastActivityDate: 2017-06-25T18:22:53.127Z
Answers.UserDisplayName: Ellen Spertus
Answers.Body: <p><em>TL;DR: If there's no special reason against it, release them before the assignments are done. If you'd like to keep them secret to the students while developing, I would still recommend releasing them after the assignment for the students to help them to evaluate their mistakes.</em></p>
  
  <hr>
  
  <p>I think whether and when the test cases should be released depends on a few factors.</p>
  
  <h2>Not releasing the test cases</h2>
  
  <p>You say that the assignments were only partially assessed by the unit tests, so in my understanding this means that there is somebody having a look at the source code. </p>
  
  <p>If (and only if) the assessment was fully automated and there were no human beings looking at the source code, you should not release the test cases before the assignment as otherwise it would be possible that somebody just created a program that has the required outputs for the test cases hard-coded and chooses the right one (using some kind of map or dictionary) depending on the input.</p>
  
  <p>If you plan to put the same (auto-graded) assignment again on students in e.g. multiple courses, you might also want to avoid releasing the test cases after the assignment - otherwise students that have completed the assignment could be able to hand the required test cases through to the students that haven't yet completed the assignment.</p>
  
  <p>If you however change the unit tests for an auto-graded system after releasing them (i.e. each course has it's own set of unit tests for the same task), then you could release them after the assignment.</p>
  
  <h2>Releasing the test cases before the assignment</h2>
  
  <p>I can imagine a situation where it's neccessary to release the test cases and to do this before the assignment: If you're doing <a href="https://en.wikipedia.org/wiki/Test-driven_development" rel="noreferrer">test-driven development</a>. In this case having all the unit tests available to everybody would be essential as otherwise the whole strategy wouldn't work.</p>
  
  <p>Furthermore, the test cases could help students as they are examples of the program's in- and output if they have trouble understanding the specification (as specifications can benefit from examples).</p>
  
  <p>I would even go further and would recommend you to release the test cases before the assignment in almost any case unless you have specific reasons against it.</p>
  
  <h2>Releasing it after the assignment</h2>
  
  <p>This could be an option if you want the students to figure out the algorithm without having examples (not sure when this is required, but maybe there are some sort of tasks that require that) but you want them to give a chance to review their mistakes after the assignment is done. I would always try to release them at the latest when the assignment is done.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-25T18:39:14.573Z
Answers.LastActivityDate: 2017-06-25T18:39:14.573Z
Answers.UserDisplayName: TuringTux
Answers.Body: <p>I've always released a number of test cases for the purposes of clarity. I do have to double- and triple-check that my unpublished test cases are nevertheless <em>unambiguously</em> specified in the documentation I provided.</p>
  
  <p>However, I am a computer science teacher, not a mystery novelist. It would never occur to me to deduct points for test cases that are not provided in advance without also telling the student what the test case in question is at the moment of the deduction. That seems plum crazy to me; how would the students learn this way? </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-25T18:25:49.010Z
Answers.Text: "That seems plum crazy to me": It's one of a few things that has frustrated me with my department..
Answers.UserDisplayName: Adam Williams
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-25T18:19:44.417Z
Answers.LastActivityDate: 2017-06-25T18:19:44.417Z
Answers.UserDisplayName: Ben I.
Answers.Body: <p>The only time I would not provide them, is if you are asking the students to derive test cases. You should not be testing to see if they can guess the test cases. Success criteria, should always be made available. </p>
  
  <p>Exceptions, have most test cases available to pupils, but have one or two extra, but ensure that there is enough info in the question to derive it. One of the secret tests should be to test big O of the solution (as codility do on their on-line tests. They seem to specify the big-O for every question, but not the test).</p>
  
  <p>Afterwards make it very clear where they lost grades (codility does not public the code for their tests, but does tell you what it tests, e.g. failed test of <code>O(n²)</code>. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-25T18:29:27.547Z
Answers.Text: Not sure how I feel about the secret time complexity testing. I'd be okay with it if the spec explicitly asked for the most time-efficient algorithm to be implemented, but there's often a trade-off between space-complexity, time-complexity and hell .. readability. Should students be penalised for prioritising the "wrong" thing there?
Answers.UserDisplayName: Adam Williams
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-25T18:38:47.837Z
Answers.Text: @AdamWilliams yes specify big-O in the question. Unless it in implicit e.g. implement quick sort. And while I would always say optimise for readability, I would also say that minimising complexity is making it more readable. Having a loop within a loop when one loop will do is not cool, but having a loop follow a loop is ok. Having more code that me in the loop is also ok. Though we could ask, could that be moved out of the loop, it would not fail a big-O test.
Answers.UserDisplayName: ctrl-alt-delor
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-25T21:09:20.430Z
Answers.Text: "You should not be testing to see if they can guess the test cases." But you **should** be testing to see if they can write correct programs. "Not penalizing them for not deriving their own test cases" amounts to "not penalizing them for not dong the work properly" - unless the objective of the course is to produce graduates who can't manage "fizzbuzz" type questions in job interviews!
Answers.UserDisplayName: alephzero
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T17:19:10.730Z
Answers.Text: @alephzero you are infinity confusing, please cite all of your quotations. If we do not have a link to them how can we know what your point is.
Answers.UserDisplayName: ctrl-alt-delor
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-25T18:20:23.127Z
Answers.LastActivityDate: 2017-06-25T18:33:52.103Z
Answers.LastEditDate: 2017-06-25T18:33:52.103Z
Answers.UserDisplayName: ctrl-alt-delor
Answers.Body: <p><strong>Yes, Absolutely</strong></p>
  
  <p>Test cases are necessary to communicate what you expect the student's program to do.</p>
  
  <p>Usually, instructors create 3-5 test cases and make at least 2 visible to the students. That way, students know what is expected but can't tailor their solution to just those test cases as they still need to accommodate for the hidden ones.</p>
  
  <p>Test cases must include:</p>
  
  <ul>
  <li>Exact stdout (output printed on the console)</li>
  <li>Clear indication of what will be user input using stdin</li>
  <li>Any files available to the student to read from (remember to tell the students what the name of the file is and the formatting of it)</li>
  <li>Any files that the student is expected to write (tell them what path they should write to and how they should format their files)</li>
  </ul>
  
  <p>Here's an example of a test case that is provided to students on <a href="http://www.autogradr.com" rel="noreferrer">AutoGradr</a>:
  <a href="https://i.sstatic.net/0nUIe.png" rel="noreferrer"><img src="https://i.sstatic.net/0nUIe.png" alt="Test Cases on AutoGradr.com"></a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T17:42:10.083Z
Answers.Text: Very nice answer - I like the emphasis on clarity of the assignment itself. Welcome to [cseducators.se]! I hope we hear more from you in the future :)
Answers.UserDisplayName: Ben I.
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T17:40:25.213Z
Answers.LastActivityDate: 2017-06-26T17:40:25.213Z
Answers.UserDisplayName: tusharsoni
Answers.Body: <p>Test cases are a great tool for students relatively new to the discipline. Since I teach CS50 AP, my students have access to the command-line tool <code>check50</code>, which they can run for nearly all the programs they write at the beginning of the year.</p>
  
  <p>For my students who are new to the discipline, this helps them understand the need to read the problem specification thoroughly since each test case connects directly to something they are asked to implement. It's a relatively simple tool: if a test passes, they'll see a green <code>:)</code>; if it fails, a red <code>:(</code>; if something unexpected, a yellow <code>:|</code>. </p>
  
  <p>As the difficult and the size of the programs scale, students have to stop relying on the tool and instead have to test their code thoroughly based on a careful, thorough reading of <em>exactly</em> what they are being asked to do. Scaffolding in this way shifts the onus over the course of the year: they are given each test case in the beginning, but as the year progresses, it is more and more their responsibility to test their code for all possible inputs. This gradual transition works well. Here's how CS50 explains it when students reach the first assignment -- one which uses random numbers -- for which there is no <code>check50</code>:</p>
  
  <blockquote>
    <p>When ready to check the correctness of your program officially with
    <code>check50</code>...well, you can’t. Reason being that the way the staff
    solution generates random numbers might in fact be different from your
    own, even though both do properly generate sets of random numbers.
    It’s up to you to determine that your program produces:</p>
    
    <ul>
    <li><p>the correct number of pseudorandomly generated numbers,</p></li>
    <li><p>each of which is greater than or equal to 0 and also less than (and
    not equal to) <code>max</code>, and</p></li>
    <li><p>that if your program is run with the same seed value multiple times,
    the list of numbers it generates is identical from run to run.</p></li>
    </ul>
    
    <p>Not having access to <code>check50</code> for this problem is actually a good
    thing. It’s a bad idea to get into the habit of testing your code with
    <code>check50</code> before testing it yourself. (And definitely don’t get into an
    even worse habit of only testing your code with <code>check50</code>!) Suffice it
    to say <code>check50</code> doesn’t exist in the real world, so running your code
    with your own sample inputs, comparing actual output against expected
    output, is the best habit to get into sooner rather than later.</p>
    
    <p>Truly, don’t do yourself a long-term disservice!</p>
  </blockquote>
  
  <p>As a final point on this topic, <code>correctness</code> (passing each test case) is only one of four axes I use to assess their work. I also look at <code>scope</code> (attempting to implement all aspects of a program), <code>design</code> (efficiency of logic/algorithm(s) in a program), and <code>style</code> (readability vis-a-vis variable names, indentation, etc.). If you only grade on getting the code right, then I could see not providing test cases for every test case. However, if there is a bigger picture to assess in terms of code style and design, then I see no problem in giving as much information up front in order to aid student learning.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-25T20:50:40.587Z
Answers.LastActivityDate: 2017-06-25T20:50:40.587Z
Answers.UserDisplayName: Peter
Answers.Body: <p>I would say that all test cases <strong>don't</strong> need to be given, as long as the tests are not the only grading component. <br><br> If a teacher gives clear instructions for what a program should and should not do without specific test cases, students have to be able to come up with their own test cases which is good experience. Programming for test cases is never a good idea, and if that is what students are focusing on, it can lead to shortcuts or lazy code. <br><br> If a program fails a test case, the input or actions that failed should be given. Then a student has to run their program themselves to find out why it is failing. If given the test case instead, it is usually more obvious what was being tested for and therefore easier to find the error (which is less debugging experience).</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T19:29:23.267Z
Answers.LastActivityDate: 2017-06-26T19:29:23.267Z
Answers.UserDisplayName: Confuzing
Answers.Body: <p>I think you should release test cases which test all of the functionality you want, but have private tests which test different different but similar inputs.</p>
  
  <p>This ensures students have not just poked their code until exactly the tests have passed and then stopped. However, I find that when students are not given a test for functionality in advance they will often fail because they interpreted the spec differently -- it is in my experience extremely hard to pin down every corner in a specification, while saying to students "if in doubt, pass the tests" often helps clear up many corner cases.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-01T19:55:50.060Z
Answers.LastActivityDate: 2017-07-01T19:55:50.060Z
Answers.UserDisplayName: Chris Jefferson
Body: <p>A number of the assignments I've completed during the course of my undergraduate degree have been assessed partially using automated tests. These have never been available to students prior to the assignment submission, but I know this isn't the case at some universities. It would seem difficult to me to get less than 100% if all the test cases used for grading were released prior to the assignment deadline (provided the student gave enough time to the assignment).</p>
  
  <p>In some cases, the feedback I've received has indicated that my submission failed a number of automated tests - but the test cases were never made available to students and I was left unsure as to what I had done wrong in my work and how I could improve next time.</p>
  
  <p>In other cases, the test cases were made available <em>after</em> the assignment grades were released. I much preferred this approach, because I could see exactly where I'd gone wrong and the whole process felt fairer.</p>
  
  <p>Should automated test cases be released and if so, at what point should they be released? </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-26T16:22:35.110Z
Comments.Text: This question also relates to the discussion on providing solutions [https://cseducators.stackexchange.com/questions/4503/should-i-provide-answers-to-exercises].
Comments.UserDisplayName: Fritz Sieker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-02T20:02:37.690Z
Comments.Text: Is the assignment for them to produce programs that they can reasonably expect to & claim to work? Then you need to first teach them how to produce such programs. So that means teaching (some form of) specification, verification (in the sense of giving justification that a program meets its specification) & testing first. Yeah, no one does this. Instead they model (I won't say "teach") hacking.
Comments.UserDisplayName: philipxy
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-25T17:31:51.993Z
Comments.Text: Depends if you are grading programming ability, or ability to guess what you are thinking.
Comments.UserDisplayName: ctrl-alt-delor
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-25T19:06:57.900Z
Comments.Text: In introductory courses at the U of Toronto, they generally incorporate a few key cases as examples in the assignment spec, and warn students that these aren't exhaustive — and then none are added that test behaviour not explicitly described in the spec. And, as @EllenSpertus says below, from the first course up there are assignments requiring test-*writing* too.
Comments.UserDisplayName: Luke Sawczak
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-25T19:09:38.163Z
Comments.Text: But I will certainly say that not all the tests in the suites I've written were given to students. As long as they're still learning their motivation is often a grade rather than a good product, so it becomes too tempting to code to the tests, including hard-coding input→output, irresponsible as it might seem. An example is a Sudoku fast-fail algorithm they needed to write. Seeing all our test cases would have resulted in uncommonly fast fails...
Comments.UserDisplayName: Luke Sawczak
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-25T19:10:53.820Z
Comments.Text: Isn't the problem of writing a test case map eliminated when instructors actually glance at the code? I'm not sure this justifies not releasing test cases, nor would I appreciate it if my grade was entirely determined by automated tests.
Comments.UserDisplayName: Adam Williams
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-25T21:13:00.400Z
Comments.Text: Releasing some *simple* test cases is a very good way for the students to check they understand what they are required to do. On the other hand, not releasing *all* the test cases used until after grading seems almost essential, otherwise the smart students will just "code to the test" rather than actually solve the problem assigned. That may get them good marks for the course, but it won't make them good real-world programmers!
Comments.UserDisplayName: alephzero
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-26T18:53:59.713Z
Comments.Text: In my experience some professors would make these tests available to students, some would describe them in technical detail in class the next day, and some would go through them individually during office hours.  I don't think I ever had a professor keep the tests a secret.  Have you tried going to the office of those who held out and asking?  Certainly as an educator I'd probably follow all three approaches simultaneously, but they are by no means obligated to do so as finding the problem creates a very difficult, but optional, exercise for the student.
Comments.UserDisplayName: Darren
Tags: <undergraduate><grading><homework>
Title: Should test cases be made available to students for assessed assignments?
ThreadText: # Should test cases be made available to students for assessed assignments?
  
  (Asked by: username_0 on 2017-06-25)
  
  A number of the assignments I've completed during the course of my undergraduate degree have been assessed partially using automated tests. These have never been available to students prior to the assignment submission, but I know this isn't the case at some universities. It would seem difficult to me to get less than 100% if all the test cases used for grading were released prior to the assignment deadline (provided the student gave enough time to the assignment).
  
  In some cases, the feedback I've received has indicated that my submission failed a number of automated tests - but the test cases were never made available to students and I was left unsure as to what I had done wrong in my work and how I could improve next time.
  
  In other cases, the test cases were made available *after* the assignment grades were released. I much preferred this approach, because I could see exactly where I'd gone wrong and the whole process felt fairer.
  
  Should automated test cases be released and if so, at what point should they be released?
  
  **Tags:** undergraduate, grading, homework
  
  > **Comments:**
  > **username_3**: Depends if you are grading programming ability, or ability to guess what you are thinking.  
  > **username_4**: In introductory courses at the U of Toronto, they generally incorporate a few key cases as examples in the assignment spec, and warn students that these aren't exhaustive — and then none are added that test behaviour not explicitly described in the spec. And, as @username_8 says below, from the first course up there are assignments requiring test-*writing* too.  
  > **username_4**: But I will certainly say that not all the tests in the suites I've written were given to students. As long as they're still learning their motivation is often a grade rather than a good product, so it becomes too tempting to code to the tests, including hard-coding input→output, irresponsible as it might seem. An example is a Sudoku fast-fail algorithm they needed to write. Seeing all our test cases would have resulted in uncommonly fast fails...  
  > **username_0**: Isn't the problem of writing a test case map eliminated when instructors actually glance at the code? I'm not sure this justifies not releasing test cases, nor would I appreciate it if my grade was entirely determined by automated tests.  
  > **username_5**: Releasing some *simple* test cases is a very good way for the students to check they understand what they are required to do. On the other hand, not releasing *all* the test cases used until after grading seems almost essential, otherwise the smart students will just "code to the test" rather than actually solve the problem assigned. That may get them good marks for the course, but it won't make them good real-world programmers!  
  > **username_6**: In my experience some professors would make these tests available to students, some would describe them in technical detail in class the next day, and some would go through them individually during office hours.  I don't think I ever had a professor keep the tests a secret.  Have you tried going to the office of those who held out and asking?  Certainly as an educator I'd probably follow all three approaches simultaneously, but they are by no means obligated to do so as finding the problem creates a very difficult, but optional, exercise for the student.  
  > **username_1**: This question also relates to the discussion on providing solutions [https://cseducators.stackexchange.com/questions/4503/should-i-provide-answers-to-exercises].  
  > **username_2**: Is the assignment for them to produce programs that they can reasonably expect to & claim to work? Then you need to first teach them how to produce such programs. So that means teaching (some form of) specification, verification (in the sense of giving justification that a program meets its specification) & testing first. Yeah, no one does this. Instead they model (I won't say "teach") hacking.  
  
  ---
  
  ## 9 answers
  
  ### Answer by username_8 on 2017-06-25. Score: 50 | Accepted answer
  
  My best practice would be to provide students with test cases and require them to submit additional test cases with their code. Then run everyone's test cases against everyone's code. Let the students know that they will get extra credit if their tests break anyone's code. That will put them in a mindset of creating tough tests and writing bullet-proof code, rather than stopping when the pass the provided tests.
  
  > **Comments:**
  > **username_0**: I really like this solution, seems like a great way to encourage students to think outside the box and write tests for situations that you may not have thought of initially.  
  > **username_9**: But make sure the student-provided test cases *should* pass first.  
  > **username_10**: A course at my Uni did this! There were two stages: At first we had to upload code that would be automatically tested against testcases and in the second stage we would upload testcases to run against the other students solutions. It was very fun to work against the testcases, but the second stage was to easy as most students had not beat many testcases from the first stage and also the objective wasn't clear (do we have to pass our own cases?). Also as @immibis said you should make sure that the test cases are correct by running them against a reference solution.  
  > **username_11**: I feel like you would get too many badly written test cases that fail when they shouldn't.  At the least have what you consider to be a very well written sample version of the code that you run the test cases against.  I'd probably ignore every test case that fails a valid sample code.  Sure someone may have written a test case that actually caught a bug in your sample 'working' implementation, but most of the time they won't and it's not worth the effort of identifying which tests are valid and which aren't.  
  > **username_12**: @username_10 Are bad tests flushed out by testing them against a canonical implementation of the program?  
  > **username_10**: @username_12 In the Uni course I was talking about bad tests weren't 'flushed out', but you were limited to two tests and your implementation had to pass at least one of these. You were encouraged  to write cases that your own program would pass, so that your program would pass more cases. But I think it would have been better to check the cases against a reference implementation and flushing them out.  
  > **username_12**: @username_10 Makes sense. It also is a good lesson in that you need to have good tests before you have good testing. Sometimes people forget that tests are not always right and not always good forever - sometimes you "break a test" and the functionality has actually changed. Quality is a goal, "making tests pass" never should be!  
  > **username_13**: When I have tried having students submit their own tests, they have been of very low quality in my experience. Often they are testing tiny corner cases which where there was some issue in the original spec.  It's VERY hard to write specs where you cover every possible corner case, particular when it comes to things like what happens when integers overflow, or code runs out of memory.  
  
  ---
  
  ### Answer by username_15 on 2017-06-25. Score: 18
  
  I've always released a number of test cases for the purposes of clarity. I do have to double- and triple-check that my unpublished test cases are nevertheless *unambiguously* specified in the documentation I provided.
  
  However, I am a computer science teacher, not a mystery novelist. It would never occur to me to deduct points for test cases that are not provided in advance without also telling the student what the test case in question is at the moment of the deduction. That seems plum crazy to me; how would the students learn this way?
  
  > **Comments:**
  > **username_0**: "That seems plum crazy to me": It's one of a few things that has frustrated me with my department..  
  
  ---
  
  ### Answer by username_14 on 2017-06-25. Score: 10
  
  *TL;DR: If there's no special reason against it, release them before the assignments are done. If you'd like to keep them secret to the students while developing, I would still recommend releasing them after the assignment for the students to help them to evaluate their mistakes.*
  
  ---
  
  I think whether and when the test cases should be released depends on a few factors.
  
  ## Not releasing the test cases
  
  You say that the assignments were only partially assessed by the unit tests, so in my understanding this means that there is somebody having a look at the source code.
  
  If (and only if) the assessment was fully automated and there were no human beings looking at the source code, you should not release the test cases before the assignment as otherwise it would be possible that somebody just created a program that has the required outputs for the test cases hard-coded and chooses the right one (using some kind of map or dictionary) depending on the input.
  
  If you plan to put the same (auto-graded) assignment again on students in e.g. multiple courses, you might also want to avoid releasing the test cases after the assignment - otherwise students that have completed the assignment could be able to hand the required test cases through to the students that haven't yet completed the assignment.
  
  If you however change the unit tests for an auto-graded system after releasing them (i.e. each course has it's own set of unit tests for the same task), then you could release them after the assignment.
  
  ## Releasing the test cases before the assignment
  
  I can imagine a situation where it's neccessary to release the test cases and to do this before the assignment: If you're doing [test-driven development](https://en.wikipedia.org/wiki/Test-driven_development). In this case having all the unit tests available to everybody would be essential as otherwise the whole strategy wouldn't work.
  
  Furthermore, the test cases could help students as they are examples of the program's in- and output if they have trouble understanding the specification (as specifications can benefit from examples).
  
  I would even go further and would recommend you to release the test cases before the assignment in almost any case unless you have specific reasons against it.
  
  ## Releasing it after the assignment
  
  This could be an option if you want the students to figure out the algorithm without having examples (not sure when this is required, but maybe there are some sort of tasks that require that) but you want them to give a chance to review their mistakes after the assignment is done. I would always try to release them at the latest when the assignment is done.
  
  
  ---
  
  ### Answer by username_3 on 2017-06-25. Score: 5
  
  The only time I would not provide them, is if you are asking the students to derive test cases. You should not be testing to see if they can guess the test cases. Success criteria, should always be made available.
  
  Exceptions, have most test cases available to pupils, but have one or two extra, but ensure that there is enough info in the question to derive it. One of the secret tests should be to test big O of the solution (as codility do on their on-line tests. They seem to specify the big-O for every question, but not the test).
  
  Afterwards make it very clear where they lost grades (codility does not public the code for their tests, but does tell you what it tests, e.g. failed test of `O(n²)`.
  
  > **Comments:**
  > **username_0**: Not sure how I feel about the secret time complexity testing. I'd be okay with it if the spec explicitly asked for the most time-efficient algorithm to be implemented, but there's often a trade-off between space-complexity, time-complexity and hell .. readability. Should students be penalised for prioritising the "wrong" thing there?  
  > **username_3**: @username_0 yes specify big-O in the question. Unless it in implicit e.g. implement quick sort. And while I would always say optimise for readability, I would also say that minimising complexity is making it more readable. Having a loop within a loop when one loop will do is not cool, but having a loop follow a loop is ok. Having more code that me in the loop is also ok. Though we could ask, could that be moved out of the loop, it would not fail a big-O test.  
  > **username_5**: "You should not be testing to see if they can guess the test cases." But you **should** be testing to see if they can write correct programs. "Not penalizing them for not deriving their own test cases" amounts to "not penalizing them for not dong the work properly" - unless the objective of the course is to produce graduates who can't manage "fizzbuzz" type questions in job interviews!  
  > **username_3**: @username_5 you are infinity confusing, please cite all of your quotations. If we do not have a link to them how can we know what your point is.  
  
  ---
  
  ### Answer by username_16 on 2017-06-26. Score: 5
  
  **Yes, Absolutely**
  
  Test cases are necessary to communicate what you expect the student's program to do.
  
  Usually, instructors create 3-5 test cases and make at least 2 visible to the students. That way, students know what is expected but can't tailor their solution to just those test cases as they still need to accommodate for the hidden ones.
  
  Test cases must include:
  
  * Exact stdout (output printed on the console)
  * Clear indication of what will be user input using stdin
  * Any files available to the student to read from (remember to tell the students what the name of the file is and the formatting of it)
  * Any files that the student is expected to write (tell them what path they should write to and how they should format their files)
  
  Here's an example of a test case that is provided to students on [AutoGradr](http://www.autogradr.com):
  [![Test Cases on AutoGradr.com](https://i.sstatic.net/0nUIe.png)](https://i.sstatic.net/0nUIe.png)
  
  > **Comments:**
  > **username_15**: Very nice answer - I like the emphasis on clarity of the assignment itself. Welcome to [cseducators.se]! I hope we hear more from you in the future :)  
  
  ---
  
  ### Answer by username_17 on 2017-06-25. Score: 4
  
  Test cases are a great tool for students relatively new to the discipline. Since I teach CS50 AP, my students have access to the command-line tool `check50`, which they can run for nearly all the programs they write at the beginning of the year.
  
  For my students who are new to the discipline, this helps them understand the need to read the problem specification thoroughly since each test case connects directly to something they are asked to implement. It's a relatively simple tool: if a test passes, they'll see a green `:)`; if it fails, a red `:(`; if something unexpected, a yellow `:|`.
  
  As the difficult and the size of the programs scale, students have to stop relying on the tool and instead have to test their code thoroughly based on a careful, thorough reading of *exactly* what they are being asked to do. Scaffolding in this way shifts the onus over the course of the year: they are given each test case in the beginning, but as the year progresses, it is more and more their responsibility to test their code for all possible inputs. This gradual transition works well. Here's how CS50 explains it when students reach the first assignment -- one which uses random numbers -- for which there is no `check50`:
  
  > When ready to check the correctness of your program officially with
  > `check50`...well, you can’t. Reason being that the way the staff
  > solution generates random numbers might in fact be different from your
  > own, even though both do properly generate sets of random numbers.
  > It’s up to you to determine that your program produces:
  >
  > * the correct number of pseudorandomly generated numbers,
  > * each of which is greater than or equal to 0 and also less than (and
  >   not equal to) `max`, and
  > * that if your program is run with the same seed value multiple times,
  >   the list of numbers it generates is identical from run to run.
  >
  > Not having access to `check50` for this problem is actually a good
  > thing. It’s a bad idea to get into the habit of testing your code with
  > `check50` before testing it yourself. (And definitely don’t get into an
  > even worse habit of only testing your code with `check50`!) Suffice it
  > to say `check50` doesn’t exist in the real world, so running your code
  > with your own sample inputs, comparing actual output against expected
  > output, is the best habit to get into sooner rather than later.
  >
  > Truly, don’t do yourself a long-term disservice!
  
  As a final point on this topic, `correctness` (passing each test case) is only one of four axes I use to assess their work. I also look at `scope` (attempting to implement all aspects of a program), `design` (efficiency of logic/algorithm(s) in a program), and `style` (readability vis-a-vis variable names, indentation, etc.). If you only grade on getting the code right, then I could see not providing test cases for every test case. However, if there is a bigger picture to assess in terms of code style and design, then I see no problem in giving as much information up front in order to aid student learning.
  
  
  ---
  
  ### Answer by username_18 on 2017-06-26. Score: 2
  
  I would say that all test cases **don't** need to be given, as long as the tests are not the only grading component.   
    
   If a teacher gives clear instructions for what a program should and should not do without specific test cases, students have to be able to come up with their own test cases which is good experience. Programming for test cases is never a good idea, and if that is what students are focusing on, it can lead to shortcuts or lazy code.   
    
   If a program fails a test case, the input or actions that failed should be given. Then a student has to run their program themselves to find out why it is failing. If given the test case instead, it is usually more obvious what was being tested for and therefore easier to find the error (which is less debugging experience).
  
  
  ---
  
  ### Answer by username_13 on 2017-07-01. Score: 2
  
  I think you should release test cases which test all of the functionality you want, but have private tests which test different different but similar inputs.
  
  This ensures students have not just poked their code until exactly the tests have passed and then stopped. However, I find that when students are not given a test for functionality in advance they will often fail because they interpreted the spec differently -- it is in my experience extremely hard to pin down every corner in a specification, while saying to students "if in doubt, pass the tests" often helps clear up many corner cases.
  
  
  ---
  
  ### Answer by username_7 on 2017-06-26. Score: 1
  
  Some of it depends on the level of the student. There's obviously a big difference between a freshman just starting out, a senior about to graduate, and someone who's been in industry for awhile.
  
  Keep in mind that, once your students are in industry, they're going to be expected to be able to write code that passes QA, and it's usually *not* the case that QA will supply them with a bunch of unit tests in advance. This being the case, it's essential that people learn to test their own stuff before they graduate.
  
  I'm of the opinion that students should be taught how to write proper unit tests as early as possible. This should eventually progress to other kinds of automated testing over time as appropriate, such as Selenium or Coded UI tests. It's also useful to be aware of major testing techniques and terminology, such as [boundary value analysis](https://en.wikipedia.org/wiki/Boundary-value_analysis).
  
  With this said, it's probably helpful to give new students tests early on and then slowly "wean" them off of that.
  
  When I was in college, some of my instructors would also suggest test cases as part of the assignment.
  
  I also agree with @username_8's suggestions for having students submit test cases as part of the assignment.
  
  For learning purposes, you could start students out with "here's a piece of code, suggest 5 test cases for this" or "here's some code, write some unit tests for it." Again, you could start with giving them specific things to test for and progressively decreasing the amount you give them "up front" with the goal of eventually getting to "give me whatever automated tests you think are reasonable with your assignment" and grading them on how good and thorough the tests are.
  
  
  ---
  
  