Answers.Body: <p>Rather than just try to motivate them, you can change the structure of the assignment. Testing on family members and friends seems like it is the problem.</p>
  <p>Instead, break the assignment into two parts: building and testing. Teams build their own product and, at some point, are matched with another team who's assignment is to test the first team's work. Both parts are graded; the quality of the build and the quality of the test you provide for the other team you were matched with.</p>
  <p>So, team A's product is tested by team B. Team B's product is tested by team C (which might be A, or not). Team B gets a grade for its own product and on the quality of the feedback provided to A.</p>
  <p>You will have to explain some things about the testing part. The purpose is not to tear down the other team, but to provide them feedback on what needs fixing. This can also provide a way to encourage writing along with development, which is something many students need to practice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T14:41:30.790Z
Answers.Text: Right. It's the old horse race technique: have them switch horses.
Answers.UserDisplayName: Scott Rowe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-29T12:03:30.600Z
Answers.Text: Switching teams is a good idea, but I still have the problem of how to verify that the testing team really tests the product, and not just makes up feedback that looks deep?
Answers.UserDisplayName: Erel Segal-Halevi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-29T12:12:18.663Z
Answers.Text: If I remember correctly from past interactions, you have a very difficult scale to work with. Too many students, too little help. If you make assignments then you need a way to properly evaluate them. It ain't easy, as you know but scale can make it impossible.
Answers.UserDisplayName: Buffy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-31T14:13:30.330Z
Answers.Text: Even in small scale, I do not see how I can distinguish between genuine feedback, and feedback made up for the exercise?
Answers.UserDisplayName: Erel Segal-Halevi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-31T14:30:01.843Z
Answers.Text: You have to read what they write and evaluate it. There isn't any magic in it. Though I fear that magic is required at your scale.
Answers.UserDisplayName: Buffy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-04T22:56:56.597Z
Answers.Text: A group of students — technically minded _and_ familiar with the particular problem — are hardly representative users, and much less likely to spot issues with usability, especially the higher-level ones around understanding and interacting with the models and metaphors used. (Of course, they're _more_ likely to spot bugs and similar, so that can still be very useful testing. But no substitute for real end-users.)
Answers.UserDisplayName: gidds
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T12:08:34.763Z
Answers.LastActivityDate: 2022-07-27T12:08:34.763Z
Answers.UserDisplayName: Buffy
Answers.Body: <p>You're at something of a dead end.  The problem is intrinsic to the assignment; they are aware that, once their project is done, very little is likely to come of it.  Thus, it isn't worth putting real, hefty effort into obtaining serious feedback -- to what practical end?</p>
  <p>You say that most students find the exercise worthwhile enough to engage in, but those who do not are not being irrational actors.</p>
  <p>You can try to incentivize this with grades, but I would suggest that all this would do is cause those who already are disinterested to take a bit more trouble to cover up their tracks.</p>
  <p>Again, if I were in your position, I wouldn't lean into the grade aspect.  That feels like force, and is fighting against what you want to achieve, which is intrinsic motivation.</p>
  <p>What you want, of course, is for them to see the value in it and take it seriously.  So what, then, is the value proposition for the students?  If I were in your position, I would begin there.</p>
  <p>Motivation starts from within, so if you want serious work on obtaining feedback, you need to explain why this part is important.  You can say that it is particularly important to <em>you</em>, and that will move the needle somewhat (it really will!), but you must also undertake the comparatively harder effort of explaining what you hope the students will get out of this portion if they take it seriously, why you believe it will be valuable to them, and thus the approach that they should take.</p>
  <p>When the students see value in it, you will see the results.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-29T12:06:05.093Z
Answers.Text: I do explain the internal importance of this exercise. However, the students have other simultaneous courses that give them exercises with both internal importance *and grades*. When students has to choose where to put their time and effort, the grade aspect wins.
Answers.UserDisplayName: Erel Segal-Halevi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T03:32:13.550Z
Answers.LastActivityDate: 2022-07-27T03:32:13.550Z
Answers.UserDisplayName: Ben I.
Body: <p>I advise several teams of undergraduate students in their final project. It is an annual softward-development project that they do during their 3rd year of studies. Most of them develop a web application in a topic they choose. The development proceeds in several phases. One of the final phases, after the application itself is mostly ready, is to give the application to end users (e.g. family and friends), watch them use the application, ask for their feedback, and prepare a report on issues such as: ease of use, enjoyment, user interface, unexpected errors, etc. This report is 10% of the final grade.</p>
  <p>Most teams do a pretty good job, but some teams submit a report with generic answers, and I have a gut feeling that they made it up  (or, at least, did not do much effort to elicit substantial feedback from the users). This is quite understandable, since they do not intend to bring this application to the market - they just want to get the grade and proceed with their other chores.</p>
  <p>My question: how can I incentivize students to take this exercise more seriously?</p>
  <p>One thing I considered is: after they submit the report, I can test the system by my own. For every bug that I find, deduct 10% of the grade for this phase. This will incentivize them to test the system thouroughly for bugs. But this does not handle other usability issues that are not proper bugs.</p>
  <p>What else can I do to incentivize students to test their system with end-users?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-09T07:06:13.970Z
Comments.Text: From my experience as one of the students that did care, I took this kind of course as a serious learning opportunity. Testing from real users wasn't even required, but one of my teammates and I took it so far as to get a programming youtuber to review our open source software in a real stream, and we got to discuss not just with the youtuber but also the audience as everyone was trying out our code. It was an amazing learning experience that I'm glad to have been able to taken, and I understand why you would want to incentivize students to do the same. (cont.)
Comments.UserDisplayName: Simply Beautiful Art
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-09T07:12:05.687Z
Comments.Text: (cont.) However, I think it is worth pointing out that in the process of trying to get people who don't want to do this, you may hinder those that actually do. For example, @Buffy's approach can waste their time if they actually want to test their software with a different audience, and their peers may honestly not be good test subjects for things like the usability issues you mentioned. What I mean to say is that for those that actually want to do this, it may be better to be less restrictive and instead provide options or examples from previous semesters that show the before/after difference
Comments.UserDisplayName: Simply Beautiful Art
Tags: <project><grading><unit-testing>
Title: How can I incentivize students to test their applications with end users?
ThreadText: # How can I incentivize students to test their applications with end users?
  
  (Asked by: username_0 on 2022-07-26)
  
  I advise several teams of undergraduate students in their final project. It is an annual softward-development project that they do during their 3rd year of studies. Most of them develop a web application in a topic they choose. The development proceeds in several phases. One of the final phases, after the application itself is mostly ready, is to give the application to end users (e.g. family and friends), watch them use the application, ask for their feedback, and prepare a report on issues such as: ease of use, enjoyment, user interface, unexpected errors, etc. This report is 10% of the final grade.
  
  Most teams do a pretty good job, but some teams submit a report with generic answers, and I have a gut feeling that they made it up (or, at least, did not do much effort to elicit substantial feedback from the users). This is quite understandable, since they do not intend to bring this application to the market - they just want to get the grade and proceed with their other chores.
  
  My question: how can I incentivize students to take this exercise more seriously?
  
  One thing I considered is: after they submit the report, I can test the system by my own. For every bug that I find, deduct 10% of the grade for this phase. This will incentivize them to test the system thouroughly for bugs. But this does not handle other usability issues that are not proper bugs.
  
  What else can I do to incentivize students to test their system with end-users?
  
  **Tags:** project, grading, unit-testing
  
  > **Comments:**
  > **username_1**: From my experience as one of the students that did care, I took this kind of course as a serious learning opportunity. Testing from real users wasn't even required, but one of my teammates and I took it so far as to get a programming youtuber to review our open source software in a real stream, and we got to discuss not just with the youtuber but also the audience as everyone was trying out our code. It was an amazing learning experience that I'm glad to have been able to taken, and I understand why you would want to incentivize students to do the same. (cont.)  
  > **username_1**: (cont.) However, I think it is worth pointing out that in the process of trying to get people who don't want to do this, you may hinder those that actually do. For example, @username_2's approach can waste their time if they actually want to test their software with a different audience, and their peers may honestly not be good test subjects for things like the usability issues you mentioned. What I mean to say is that for those that actually want to do this, it may be better to be less restrictive and instead provide options or examples from previous semesters that show the before/after difference  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-07-27. Score: 6
  
  Rather than just try to motivate them, you can change the structure of the assignment. Testing on family members and friends seems like it is the problem.
  
  Instead, break the assignment into two parts: building and testing. Teams build their own product and, at some point, are matched with another team who's assignment is to test the first team's work. Both parts are graded; the quality of the build and the quality of the test you provide for the other team you were matched with.
  
  So, team A's product is tested by team B. Team B's product is tested by team C (which might be A, or not). Team B gets a grade for its own product and on the quality of the feedback provided to A.
  
  You will have to explain some things about the testing part. The purpose is not to tear down the other team, but to provide them feedback on what needs fixing. This can also provide a way to encourage writing along with development, which is something many students need to practice.
  
  > **Comments:**
  > **username_3**: Right. It's the old horse race technique: have them switch horses.  
  > **username_0**: Switching teams is a good idea, but I still have the problem of how to verify that the testing team really tests the product, and not just makes up feedback that looks deep?  
  > **username_2**: If I remember correctly from past interactions, you have a very difficult scale to work with. Too many students, too little help. If you make assignments then you need a way to properly evaluate them. It ain't easy, as you know but scale can make it impossible.  
  > **username_0**: Even in small scale, I do not see how I can distinguish between genuine feedback, and feedback made up for the exercise?  
  > **username_2**: You have to read what they write and evaluate it. There isn't any magic in it. Though I fear that magic is required at your scale.  
  > **username_4**: A group of students — technically minded _and_ familiar with the particular problem — are hardly representative users, and much less likely to spot issues with usability, especially the higher-level ones around understanding and interacting with the models and metaphors used. (Of course, they're _more_ likely to spot bugs and similar, so that can still be very useful testing. But no substitute for real end-users.)  
  
  ---
  
  ### Answer by username_5 on 2022-07-27. Score: 3
  
  You're at something of a dead end. The problem is intrinsic to the assignment; they are aware that, once their project is done, very little is likely to come of it. Thus, it isn't worth putting real, hefty effort into obtaining serious feedback -- to what practical end?
  
  You say that most students find the exercise worthwhile enough to engage in, but those who do not are not being irrational actors.
  
  You can try to incentivize this with grades, but I would suggest that all this would do is cause those who already are disinterested to take a bit more trouble to cover up their tracks.
  
  Again, if I were in your position, I wouldn't lean into the grade aspect. That feels like force, and is fighting against what you want to achieve, which is intrinsic motivation.
  
  What you want, of course, is for them to see the value in it and take it seriously. So what, then, is the value proposition for the students? If I were in your position, I would begin there.
  
  Motivation starts from within, so if you want serious work on obtaining feedback, you need to explain why this part is important. You can say that it is particularly important to *you*, and that will move the needle somewhat (it really will!), but you must also undertake the comparatively harder effort of explaining what you hope the students will get out of this portion if they take it seriously, why you believe it will be valuable to them, and thus the approach that they should take.
  
  When the students see value in it, you will see the results.
  
  > **Comments:**
  > **username_0**: I do explain the internal importance of this exercise. However, the students have other simultaneous courses that give them exercises with both internal importance *and grades*. When students has to choose where to put their time and effort, the grade aspect wins.  
  
  ---
  
  