Answers.Body: <p>I don't think you can just by using an auto grader.</p>
  
  <p>I use an auto grader to tell me if the code solves the problem it's supposed to solve. And I tell my students that any code that solves the given problem is correct. Some people write a 2 line solution, some write a 50 line solution. But I don't consider either one more correct than the other. Generally the shorter solution takes more mental work and the longer solution takes more physical work (typing). </p>
  
  <p>But I will go back and look at the submissions. Since the auto grader has checked it, I don't have to worry about whether the code builds or gets the right answer. That takes out the time consuming, boring part for me and lets me focus on style and any tips that I can give. </p>
  
  <p>I guess, especially early in the year, I don't want to penalize students for a working solution that might not be as clean or elegant as another student's. There's enough frustration in a first year course without worrying too much about style and efficiency.  </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T22:32:01.393Z
Answers.Text: Out of curiosity, which auto grader tool do you use?
Answers.UserDisplayName: tusharsoni
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-27T01:55:48.913Z
Answers.Text: @tusharsoni Actually wrote my own. Link is in my profile.
Answers.UserDisplayName: Ryan Nutt
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-20T19:35:49.800Z
Answers.Text: "There's enough frustration in a first year course without worrying too much about style and efficiency." The problem here is that when bad habits get ingrained early on, it much more work and frustration to fix them later.
Answers.UserDisplayName: Daniel R. Collins
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-28T18:28:58.600Z
Answers.LastActivityDate: 2017-05-28T18:28:58.600Z
Answers.UserDisplayName: Ryan Nutt
Answers.Body: <p>I am a strong advocate for <em>code interviews</em>.  I make principles of style into about 50% of a lab grade (though I do permit kids to go back and refactor after an interview for full credit.)</p>
  
  <p>For some context, I discuss regularly that there are <em>two audiences of code:</em> computers, and people, and that these audiences have very different needs.</p>
  
  <p>Since my high school class is structured with lab periods, I will grade lab 1 during lab 2 work times (and so on). This gives me class time to get my interviews done. At the start, the code needs a lot of work. However, by the time we have arrived at lab 3, the kids are <em>mostly</em> creating crystal clear code.</p>
  
  <p>Don't drop that autograder! Even with code interviews, it is still tremendously useful. Having a good autograder means that, during the interview, I <em>only</em> need to look for style issues. I can assume that even the strangest of their code (and boy, is there ever strange code at the beginning!) produces the correct output.  That permits the interview process to focus in, laser-like, on how well the student wrote for the second audience.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-05T20:39:56.453Z
Answers.Text: Can't agree more. Code is for people first and foremost; otherwise, we should all have been coding 0s and 1s instead of Python and Scheme.
Answers.UserDisplayName: xuq01
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-28T13:16:23.773Z
Answers.LastActivityDate: 2017-05-28T13:16:23.773Z
Answers.UserDisplayName: Ben I.
Answers.Body: <p>Harvard's <a href="http://cs50.wiki/file/view/Grading+CS50.pdf" rel="noreferrer">grading policy for CS50</a> is worth looking into. </p>
  
  <p>There are four components for the grade on problem sets (each of which involves submitting code). The overall grade is calculated as scope * (3 * correctness + 2 * design + 1 * style)</p>
  
  <p><strong>Scope</strong>: to what extent does the code implement the features required by the specification? </p>
  
  <p><strong>Correctness</strong>: to what extent is the code consistent with the specifications and free of bugs? This is done by the check50 autograder, and it's essentially unit testing. </p>
  
  <p><strong>Style</strong>: to what extent is your code readable (i.e., commented and indented with variables aptly named)? there's a component for formatting: I think in Harvard's case these marks are awarded by teaching assistants, but basic static analysis or linting might suffice. </p>
  
  <p><strong>Design</strong>: essentially, is this good code in terms of clarity, efficiency, logic, elegance - again, Harvard use TAs to award these grades, and it's hard to see a machine (or an inexperienced grader) being able to award these marks accurately any time soon. If you were determined to use automatic grading, I guess you could do something with run times for test data or the more sophisticated forms of static analysis. A compromise might be the use of peer-assessment and a detailed, criteria based rubric: peer assessment might have other benefits.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-29T07:42:07.090Z
Answers.Text: I can attest to this grading policy. As a teacher of CS50 AP, I appreciate the four-axis approach as it allows me to focus on multiple elements of students' work. It works very well, and Harvard provided great training on it before I adopted the curriculum.
Answers.UserDisplayName: Peter
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-30T17:55:03.623Z
Answers.Text: I raise my hand as well on the CS50 AP policy but I like what Choirbean said as well.  The process was very time intensive early on in the class but the programs were small so it was manageable.  Like Choirbean, I found I didn't need to keep at it all that much as most students just used better style once they understood the expectations.
Answers.UserDisplayName: Lisa Standring
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-27T03:45:02.043Z
Answers.Text: Late comment: This was very useful, and I used it as a model for how to grade assignments in my CS courses starting a few years ago. The one drawback, re: the given question, is that only 1 of the 4 components seems to be really automatable.
Answers.UserDisplayName: Daniel R. Collins
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-29T07:03:48.333Z
Answers.LastActivityDate: 2017-05-29T07:09:30.300Z
Answers.LastEditDate: 2017-05-29T07:09:30.300Z
Answers.UserDisplayName: Miles
Answers.Body: <p>First let me say that I think it is a good idea to automate as much as possible, and that there are tools that will help this.</p>
  
  <h2>The mystery test</h2>
  
  <p>However my experience with professionals is that there will always be a few that try to defeat the checker, not maliciously (they just don't get it; they are minimising error-messages, not minimising errors). So full automation may not be possible.</p>
  
  <p>Therefore:</p>
  
  <p>Add a few unpublished tests: If pupils meet the specification the test will pass, but you don't tell them the test or the result until grades are awarded.</p>
  
  <h2>Style award</h2>
  
  <p>You don't need to spend a lot of time on the manual part. If the style is good then it should make sense immediately (top marks), or mostly make sense (medium), does that work? (zero marks).</p>
  
  <p>Therefore:</p>
  
  <p>Award a style grade 0,1 or 2 based on a quick look at the code.</p>
  
  <h2>Big-O</h2>
  
  <p>I have seen on <a href="https://codility.com/programmers/lessons/1-iterations/" rel="noreferrer">codility</a> where they automate everything. There last test is always to check <a href="https://en.wikipedia.org/wiki/Big_O_notation" rel="noreferrer">big-O</a>. They run the code on a big data set, to check that run time does not increase worse then expected. They always tell the candidate what big-O to aim for.</p>
  
  <p>Therefore:</p>
  
  <p>Always have a test to check big-O. Tell pupils upfront, that order to expect.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-30T09:19:19.520Z
Answers.Text: The Big-O idea is a great one for testing that the design is reasonable automatically. Seems like a good 'sanity check' to make sure the code is scalable, which is always a good thing. Thanks for the idea!
Answers.UserDisplayName: Aurora0001
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-30T10:57:10.210Z
Answers.Text: I think codility does big-o dynamicaly (by running the code). I think that static analysis of big-o is hard to do, and get right 100%. But static complexity analysis is also good.
Answers.UserDisplayName: ctrl-alt-delor
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-30T09:12:45.780Z
Answers.LastActivityDate: 2017-05-30T09:12:45.780Z
Answers.UserDisplayName: ctrl-alt-delor
Answers.Body: <p>Adding static code analysis can be useful in determining if students have "over-engineered" the solution. While dynamic testing can show that the program produces the correct output, there are many examples of correctly working programs that I would not consider acceptable. Students can sometimes miss an important concept that you are attempting to assess, but work around their misunderstanding by adding additional code that wasn't necessary. </p>
  
  <p>I have done some testing of this by looking at code complexity (cyclomatic/McCabe and Halstead) numbers, state space, and LOC. For these values to be useful as a guide, the instructor must know expected value ranges for typical solutions to a given problem. These can be found by using exemplar solutions, either created by the instructor, or even identified from student submissions. </p>
  
  <p>In cases where correctly working submissions have been written "incorrectly", I will address the situation with the student and make sure they understand the concept that they appeared to dodge. </p>
  
  <p>If anyone's interested, I discussed this some, and included an example of this problem, in a SIGCSE paper a few years ago. If you want a copy of the paper, just email me - raymond dot pettit at acu dot edu: </p>
  
  <p>Raymond Pettit, John Homer, Roger Gee, Susan Mengel, and Adam Starbuck. 2015. An Empirical Study of Iterative Improvement in Programming Assignments. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (SIGCSE '15). ACM, New York, NY, USA, 410-415. DOI: <a href="http://dx.doi.org/10.1145/2676723.2677279" rel="nofollow noreferrer">http://dx.doi.org/10.1145/2676723.2677279</a> </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-06T19:55:49.283Z
Answers.Text: Oh, it is always wonderful when a veteran educator joins the community. Welcome to [cseducators.se]! I hope we hear more from you in the future.
Answers.UserDisplayName: Ben I.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-20T20:37:44.810Z
Answers.Text: Question: How is it that you find time in the semester for 75 assignments? I think that would be on the order of two per hour of class?
Answers.UserDisplayName: Daniel R. Collins
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-06T19:14:46.687Z
Answers.LastActivityDate: 2017-07-06T19:14:46.687Z
Answers.UserDisplayName: Ray
Answers.Body: <p>I would recommend looking into a <a href="https://stackoverflow.com/questions/49716/what-is-static-code-analysis">static code analyzer</a> or <a href="https://stackoverflow.com/questions/8503559/what-is-linting#8503586">linter</a> for whichever language you are using. These are designed to examine code for any unintelligible code and to make sure that the code follows the best practices. For example, a code analyzer or linter may suggest that you limit your line length, keep your methods small, and reduce unnecessary complexity.</p>
  
  <p>Now, while this sound very useful, this is something that you have to be careful with and configure very carefully, because some of the metrics that are used can trigger it when the code is actually fine. It may be good enough for your situation to simply give the code analyzer to your students and let them use it to optimize their code.</p>
  
  <p>If you do use a static code analyzer or linter, I'd recommend setting the limits high for how much room the students have for mistakes, because the contortions to get around a strange reason their assignment was poorly graded could make things much worse.</p>
  
  <p>From some quick research, it seems like the most widely used python linter is <a href="https://www.pylint.org/" rel="nofollow noreferrer">pylint</a>. From a quick glance at the docs, it seems to fit your goals.</p>
  
  <p>As a rubyist, I use <a href="https://github.com/bbatsov/rubocop" rel="nofollow noreferrer">rubocop</a> which is both a linter and static code analyzer.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-28T10:47:29.220Z
Answers.LastActivityDate: 2017-05-28T11:13:21.810Z
Answers.LastEditDate: 2017-05-28T11:13:21.810Z
Answers.UserDisplayName: thesecretmaster
Answers.Body: <p><strong>Using lint tools is the correct answer. But not the easy one.</strong></p>
  
  <p>A lot of people here have suggested (and correctly so) to use a static code analyzer or linter. However, these tools can overwhelm students, especially beginners, with a lot of information that they may not understand.</p>
  
  <p>I suggest using an auto grading tool that incorporates lint reports in it.</p>
  
  <p>As an example, here is a lint report that <a href="http://www.autogradr.com" rel="nofollow noreferrer">we</a> provide automatically to students after they make a submission:</p>
  
  <p><a href="https://i.sstatic.net/djr7m.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/djr7m.png" alt="Lint Report on AutoGradr.com"></a></p>
  
  <p>It gives students an overview on how they did using the star ratings. Additionally, it provides clear and actionable items on how to improve their code quality.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T21:34:33.387Z
Answers.LastActivityDate: 2017-06-26T21:37:21.297Z
Answers.LastEditDate: 2017-06-26T21:37:21.297Z
Answers.UserDisplayName: tusharsoni
Body: <p>Automatic grading is great for saving time on marking programming tasks, so there's more time to focus on actual <em>teaching</em>, but I've noticed a few flaws. Since work is essentially marked based on passing unit tests, thoughtful design often gets ignored, and excessively complex solutions are considered just as good as elegant ones.</p>
  
  <p>I recently read <a href="http://gregorulm.com/automatic-grading-of-code-submissions-in-moocs-isnt-perfect-either/" rel="noreferrer">this article</a> about how this is a problem on MITx—instead of this solution:</p>
  
  <pre><code>return max(min(hi,x),lo)
  </code></pre>
  
  <p>A student wrote this:</p>
  
  <pre><code>1 * lo + 0 * x + 0 * hi # too low
  0 * lo + 1 * x + 0 * hi # in range
  0 * lo + 0 * x + 1 * hi # too high
  </code></pre>
  
  <p>Automatic testing would find these equivalent,  but clearly one is incomprehensible and the other is relatively logical and sound.</p>
  
  <p>This is also recognised in <a href="http://www.cs.tufts.edu/~nr/cs257/archive/stephen-edwards/automated-feedback.pdf" rel="noreferrer"><em>Using Test-Driven Development in the Classroom</em></a>:</p>
  
  <blockquote>
    <p>Despite its classroom utility, an automatic grading strategy like
    the one embodied in the Curator also has a number of shortcomings:</p>
    
    <ul>
    <li>Students focus on output correctness first and foremost;
    all other considerations are a distant second at best (design,
    commenting, appropriate use of abstraction, testing one's
    own code, etc.). This is due to the fact that the most immediate
    feedback students receive is on output correctness [...]</li>
    </ul>
  </blockquote>
  
  <p>Is there any way of evaluating code style and design automatically (or at least detecting very poor solutions)? I've thought about using some static code analysis to try and evaluate solutions (e.g. <a href="https://en.wikipedia.org/wiki/Cyclomatic_complexity" rel="noreferrer">cyclomatic complexity</a>, or tools like <a href="https://github.com/mre/awesome-static-analysis" rel="noreferrer">these</a>), although I've not tried them yet. Are there any reliable solutions to this, or must it be done manually? Does static code analysis help in any way?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-28T10:23:15.420Z
Comments.Text: I would suggest solving this with programming. I would make a neural network that would evaluate the students' answers. But I feel like that's overkill. Then again, that is sometimes my style ;)
Comments.UserDisplayName: ItamarG3
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-28T10:27:19.800Z
Comments.Text: What language? For ruby there is a static code analyzer that checks for best practice called rubocop. Maybe there is a similar utility for the language of your choice
Comments.UserDisplayName: thesecretmaster
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-28T10:28:46.730Z
Comments.Text: @thesecretmaster perhaps looking into how `rubocop` works could be insightful
Comments.UserDisplayName: ItamarG3
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-28T10:34:39.850Z
Comments.Text: rubocop is a [static code analyzer](https://stackoverflow.com/questions/49716/what-is-static-code-analysis). It works by looking at the code without running it and evaluating for best practices, etc.
Comments.UserDisplayName: thesecretmaster
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-28T10:47:32.123Z
Comments.Text: @thesecretmaster More general solutions would be ideal, but Python would be the language I'm most interested in evaluating.
Comments.UserDisplayName: Aurora0001
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-28T10:51:24.750Z
Comments.Text: @subscript You should post that as an answer, since it seems like a good idea. Ideas on how to find an appropriate size limit would be interesting, too—my solution may be much shorter than theirs, but a longer solution might not be unreasonable necessarily.
Comments.UserDisplayName: Aurora0001
Tags: <grading><coding-style><auto-graders>
Title: How can I discourage poor design while using automatic grading of submissions?
ThreadText: # How can I discourage poor design while using automatic grading of submissions?
  
  (Asked by: username_0 on 2017-05-28)
  
  Automatic grading is great for saving time on marking programming tasks, so there's more time to focus on actual *teaching*, but I've noticed a few flaws. Since work is essentially marked based on passing unit tests, thoughtful design often gets ignored, and excessively complex solutions are considered just as good as elegant ones.
  
  I recently read [this article](http://gregorulm.com/automatic-grading-of-code-submissions-in-moocs-isnt-perfect-either/) about how this is a problem on MITx—instead of this solution:
  
  ```
  return max(min(hi,x),lo)
  
  ```
  
  A student wrote this:
  
  ```
  1 * lo + 0 * x + 0 * hi # too low
  0 * lo + 1 * x + 0 * hi # in range
  0 * lo + 0 * x + 1 * hi # too high
  
  ```
  
  Automatic testing would find these equivalent, but clearly one is incomprehensible and the other is relatively logical and sound.
  
  This is also recognised in [*Using Test-Driven Development in the Classroom*](http://www.cs.tufts.edu/~nr/cs257/archive/stephen-edwards/automated-feedback.pdf):
  
  > Despite its classroom utility, an automatic grading strategy like
  > the one embodied in the Curator also has a number of shortcomings:
  >
  > * Students focus on output correctness first and foremost;
  >   all other considerations are a distant second at best (design,
  >   commenting, appropriate use of abstraction, testing one's
  >   own code, etc.). This is due to the fact that the most immediate
  >   feedback students receive is on output correctness [...]
  
  Is there any way of evaluating code style and design automatically (or at least detecting very poor solutions)? I've thought about using some static code analysis to try and evaluate solutions (e.g. [cyclomatic complexity](https://en.wikipedia.org/wiki/Cyclomatic_complexity), or tools like [these](https://github.com/mre/awesome-static-analysis)), although I've not tried them yet. Are there any reliable solutions to this, or must it be done manually? Does static code analysis help in any way?
  
  **Tags:** grading, coding-style, auto-graders
  
  > **Comments:**
  > **username_1**: I would suggest solving this with programming. I would make a neural network that would evaluate the students' answers. But I feel like that's overkill. Then again, that is sometimes my style ;)  
  > **username_2**: What language? For ruby there is a static code analyzer that checks for best practice called rubocop. Maybe there is a similar utility for the language of your choice  
  > **username_1**: @username_2 perhaps looking into how `rubocop` works could be insightful  
  > **username_2**: rubocop is a [static code analyzer](https://stackoverflow.com/questions/49716/what-is-static-code-analysis). It works by looking at the code without running it and evaluating for best practices, etc.  
  > **username_0**: @username_2 More general solutions would be ideal, but Python would be the language I'm most interested in evaluating.  
  > **username_0**: @subscript You should post that as an answer, since it seems like a good idea. Ideas on how to find an appropriate size limit would be interesting, too—my solution may be much shorter than theirs, but a longer solution might not be unreasonable necessarily.  
  
  ---
  
  ## 7 answers
  
  ### Answer by username_6 on 2017-05-28. Score: 9 | Accepted answer
  
  I am a strong advocate for *code interviews*. I make principles of style into about 50% of a lab grade (though I do permit kids to go back and refactor after an interview for full credit.)
  
  For some context, I discuss regularly that there are *two audiences of code:* computers, and people, and that these audiences have very different needs.
  
  Since my high school class is structured with lab periods, I will grade lab 1 during lab 2 work times (and so on). This gives me class time to get my interviews done. At the start, the code needs a lot of work. However, by the time we have arrived at lab 3, the kids are *mostly* creating crystal clear code.
  
  Don't drop that autograder! Even with code interviews, it is still tremendously useful. Having a good autograder means that, during the interview, I *only* need to look for style issues. I can assume that even the strangest of their code (and boy, is there ever strange code at the beginning!) produces the correct output. That permits the interview process to focus in, laser-like, on how well the student wrote for the second audience.
  
  > **Comments:**
  > **username_7**: Can't agree more. Code is for people first and foremost; otherwise, we should all have been coding 0s and 1s instead of Python and Scheme.  
  
  ---
  
  ### Answer by username_8 on 2017-05-29. Score: 6
  
  Harvard's [grading policy for CS50](http://cs50.wiki/file/view/Grading+CS50.pdf) is worth looking into.
  
  There are four components for the grade on problem sets (each of which involves submitting code). The overall grade is calculated as scope \* (3 \* correctness + 2 \* design + 1 \* style)
  
  **Scope**: to what extent does the code implement the features required by the specification?
  
  **Correctness**: to what extent is the code consistent with the specifications and free of bugs? This is done by the check50 autograder, and it's essentially unit testing.
  
  **Style**: to what extent is your code readable (i.e., commented and indented with variables aptly named)? there's a component for formatting: I think in Harvard's case these marks are awarded by teaching assistants, but basic static analysis or linting might suffice.
  
  **Design**: essentially, is this good code in terms of clarity, efficiency, logic, elegance - again, Harvard use TAs to award these grades, and it's hard to see a machine (or an inexperienced grader) being able to award these marks accurately any time soon. If you were determined to use automatic grading, I guess you could do something with run times for test data or the more sophisticated forms of static analysis. A compromise might be the use of peer-assessment and a detailed, criteria based rubric: peer assessment might have other benefits.
  
  > **Comments:**
  > **username_9**: I can attest to this grading policy. As a teacher of CS50 AP, I appreciate the four-axis approach as it allows me to focus on multiple elements of students' work. It works very well, and Harvard provided great training on it before I adopted the curriculum.  
  > **username_10**: I raise my hand as well on the CS50 AP policy but I like what Choirbean said as well.  The process was very time intensive early on in the class but the programs were small so it was manageable.  Like Choirbean, I found I didn't need to keep at it all that much as most students just used better style once they understood the expectations.  
  > **username_5**: Late comment: This was very useful, and I used it as a model for how to grade assignments in my CS courses starting a few years ago. The one drawback, re: the given question, is that only 1 of the 4 components seems to be really automatable.  
  
  ---
  
  ### Answer by username_11 on 2017-05-30. Score: 5
  
  First let me say that I think it is a good idea to automate as much as possible, and that there are tools that will help this.
  
  ## The mystery test
  
  However my experience with professionals is that there will always be a few that try to defeat the checker, not maliciously (they just don't get it; they are minimising error-messages, not minimising errors). So full automation may not be possible.
  
  Therefore:
  
  Add a few unpublished tests: If pupils meet the specification the test will pass, but you don't tell them the test or the result until grades are awarded.
  
  ## Style award
  
  You don't need to spend a lot of time on the manual part. If the style is good then it should make sense immediately (top marks), or mostly make sense (medium), does that work? (zero marks).
  
  Therefore:
  
  Award a style grade 0,1 or 2 based on a quick look at the code.
  
  ## Big-O
  
  I have seen on [codility](https://codility.com/programmers/lessons/1-iterations/) where they automate everything. There last test is always to check [big-O](https://en.wikipedia.org/wiki/Big_O_notation). They run the code on a big data set, to check that run time does not increase worse then expected. They always tell the candidate what big-O to aim for.
  
  Therefore:
  
  Always have a test to check big-O. Tell pupils upfront, that order to expect.
  
  > **Comments:**
  > **username_0**: The Big-O idea is a great one for testing that the design is reasonable automatically. Seems like a good 'sanity check' to make sure the code is scalable, which is always a good thing. Thanks for the idea!  
  > **username_11**: I think codility does big-o dynamicaly (by running the code). I think that static analysis of big-o is hard to do, and get right 100%. But static complexity analysis is also good.  
  
  ---
  
  ### Answer by username_12 on 2017-07-06. Score: 4
  
  Adding static code analysis can be useful in determining if students have "over-engineered" the solution. While dynamic testing can show that the program produces the correct output, there are many examples of correctly working programs that I would not consider acceptable. Students can sometimes miss an important concept that you are attempting to assess, but work around their misunderstanding by adding additional code that wasn't necessary.
  
  I have done some testing of this by looking at code complexity (cyclomatic/McCabe and Halstead) numbers, state space, and LOC. For these values to be useful as a guide, the instructor must know expected value ranges for typical solutions to a given problem. These can be found by using exemplar solutions, either created by the instructor, or even identified from student submissions.
  
  In cases where correctly working submissions have been written "incorrectly", I will address the situation with the student and make sure they understand the concept that they appeared to dodge.
  
  If anyone's interested, I discussed this some, and included an example of this problem, in a SIGCSE paper a few years ago. If you want a copy of the paper, just email me - raymond dot pettit at acu dot edu:
  
  Raymond Pettit, John Homer, Roger Gee, Susan Mengel, and Adam Starbuck. 2015. An Empirical Study of Iterative Improvement in Programming Assignments. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (SIGCSE '15). ACM, New York, NY, USA, 410-415. DOI: <http://dx.doi.org/10.1145/2676723.2677279>
  
  > **Comments:**
  > **username_6**: Oh, it is always wonderful when a veteran educator joins the community. Welcome to [cseducators.se]! I hope we hear more from you in the future.  
  > **username_5**: Question: How is it that you find time in the semester for 75 assignments? I think that would be on the order of two per hour of class?  
  
  ---
  
  ### Answer by username_2 on 2017-05-28. Score: 3
  
  I would recommend looking into a [static code analyzer](https://stackoverflow.com/questions/49716/what-is-static-code-analysis) or [linter](https://stackoverflow.com/questions/8503559/what-is-linting#8503586) for whichever language you are using. These are designed to examine code for any unintelligible code and to make sure that the code follows the best practices. For example, a code analyzer or linter may suggest that you limit your line length, keep your methods small, and reduce unnecessary complexity.
  
  Now, while this sound very useful, this is something that you have to be careful with and configure very carefully, because some of the metrics that are used can trigger it when the code is actually fine. It may be good enough for your situation to simply give the code analyzer to your students and let them use it to optimize their code.
  
  If you do use a static code analyzer or linter, I'd recommend setting the limits high for how much room the students have for mistakes, because the contortions to get around a strange reason their assignment was poorly graded could make things much worse.
  
  From some quick research, it seems like the most widely used python linter is [pylint](https://www.pylint.org/). From a quick glance at the docs, it seems to fit your goals.
  
  As a rubyist, I use [rubocop](https://github.com/bbatsov/rubocop) which is both a linter and static code analyzer.
  
  
  ---
  
  ### Answer by username_4 on 2017-06-26. Score: 3
  
  **Using lint tools is the correct answer. But not the easy one.**
  
  A lot of people here have suggested (and correctly so) to use a static code analyzer or linter. However, these tools can overwhelm students, especially beginners, with a lot of information that they may not understand.
  
  I suggest using an auto grading tool that incorporates lint reports in it.
  
  As an example, here is a lint report that [we](http://www.autogradr.com) provide automatically to students after they make a submission:
  
  [![Lint Report on AutoGradr.com](https://i.sstatic.net/djr7m.png)](https://i.sstatic.net/djr7m.png)
  
  It gives students an overview on how they did using the star ratings. Additionally, it provides clear and actionable items on how to improve their code quality.
  
  
  ---
  
  ### Answer by username_3 on 2017-05-28. Score: 2
  
  I don't think you can just by using an auto grader.
  
  I use an auto grader to tell me if the code solves the problem it's supposed to solve. And I tell my students that any code that solves the given problem is correct. Some people write a 2 line solution, some write a 50 line solution. But I don't consider either one more correct than the other. Generally the shorter solution takes more mental work and the longer solution takes more physical work (typing).
  
  But I will go back and look at the submissions. Since the auto grader has checked it, I don't have to worry about whether the code builds or gets the right answer. That takes out the time consuming, boring part for me and lets me focus on style and any tips that I can give.
  
  I guess, especially early in the year, I don't want to penalize students for a working solution that might not be as clean or elegant as another student's. There's enough frustration in a first year course without worrying too much about style and efficiency.
  
  > **Comments:**
  > **username_4**: Out of curiosity, which auto grader tool do you use?  
  > **username_3**: @username_4 Actually wrote my own. Link is in my profile.  
  > **username_5**: "There's enough frustration in a first year course without worrying too much about style and efficiency." The problem here is that when bad habits get ingrained early on, it much more work and frustration to fix them later.  
  
  ---
  
  