Answers.Body: <p>The best way to find out is to give programming problems, or similarly complex design problems, where you can look at the details of the solutions they create. I give lots of "lab" assignments where the students have to write programs based on a problem statement from the textbook. Part of the answer is: "Are they using what we just covered in class?", but the vital part is: <strong>"Are they gaining knowledge over time and learning to assemble things with increasingly complex structure?"</strong></p>
  
  <p>Some students clearly learn general concepts like how to define classes and use them sensibly, while other students struggle with this, or even more basic concepts like how to use variables in one UI method that were set in another UI method. (Could be Console or Windows, doesn't matter.) Some students take the ideas of Relational Comparison and Boolean combiners and use them correctly, others struggle and try to create things by sticking together the code they painfully got to work in the previous assignment (which they are now trying to re-use for an opposite case).</p>
  
  <p>In short, <em>assessing</em> whether students have learned how to abstract is bone-obvious. <em>Teaching</em> it is the hard part. <strong>We need to find ways to teach abstraction.</strong> My research has turned up very little except arguments over what the word means, or worse: whether to start teaching at high levels of abstraction, or instead teach how computers actually work, etc. (See other questions on this site.) Poor start on a very significant problem!</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-07T20:45:02.117Z
Answers.Text: Welcome to CSEducators. Hope to see you again. I'm one who isn't so sure that the assessment is obvious, but agree that teaching is the hard part.
Answers.UserDisplayName: Buffy
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-06T17:39:11.810Z
Answers.LastActivityDate: 2017-10-06T17:39:11.810Z
Answers.UserDisplayName: user3243
Answers.Body: <p>tl;dr: Just say no. </p>
  
  <p>This question is difficult on many levels. I seems to me to be a land mine of misconceptions and has the possibility to lead to poor teaching practice. </p>
  
  <p>First the difficulties</p>
  
  <ol>
  <li><p>Only <em>concepts</em> can be abstract. Abstraction is about ideas. Animal is a concept. Mammal is a concept. Animal is more abstract than Mammal since it contains the idea of Mammal, but isn't restricted to it. The kitty sitting on my hearth is concrete, not less abstract than Mammal. The kitty isn't a concept it is a thing. </p></li>
  <li><p>Given some ot the examples, there is a misconception about what is abstract and what is concrete. Physical things are not abstract. One can't be <em>more</em> abstract than another. Even things that aren't physical, but which can be manipulated, are concrete, not abstract. A LinkedList (Java) isn't abstract. It is concrete. List, on the other hand (an interface) is abstract. </p></li>
  <li><p>In computing there are many, many different kinds of abstraction. Even the concept of a while loop is an abstraction. In OOP languages classes implement abstractions. Interfaces (even if the language doesn't have that explicit feature) are abstractions. So, Python has interfaces, though it has no language feature with that name. If two classes have the same interface they are realizations of the abstraction, but neither is more or less abstract than the other. If one interface contains another interface then it is more abstract.But we also have <em>data abstraction</em>. Lists and Maps define an interface (an abstraction). They can be implemented/realized many ways. Each realization is concrete, not less abstract. </p></li>
  <li><p>The web site cited seems to be trying to test everything with Multiple Choice Questions (MCQ). I question both the possibility of that and the wisdom of it. Some thing are complex MCQ are simple. Trying to push the assessment of complex things through such a small funnel seems misplaced to me - even foolish. I'm pretty sure that MCQ also advantage certain students and disadvantage others. They are fine for rote memory and facts, but pretty poor for deep understanding. I have nothing against their use unless it is overdone. MCQ are also difficult to validate. The wisdom of groups isn't enough. It takes statistical measurement and cross validation to do a good job of it. </p></li>
  <li><p>The Font Construction example given in the question also seem to be testing specific vocabulary. This example threw me completely. I think I could eliminate one answer, but also think I could justify each of the others if given the chance. If it is specific vocabulary it is just rote memory and hoping that the student "remembered" to remember the right thing. </p></li>
  <li><p>Often, getting a MCQ wrong and then later learning the <em>correct</em> answer can sometimes leave students simply puzzled and frustrated. This results in a destructive game that I'd rather not play. </p></li>
  <li><p>MCQ try to put a lot of thinking into the questions so that no one needs to put any thinking into the answers after the student finishes. This may scale well, but is a sub-optimal form of <em>education</em>. After all, it is the <em>students thoughts</em> that we want to get a handle on in assessment. </p></li>
  </ol>
  
  <p>I don't believe that a single, or even a small number of MCQ could capture the student's understanding of abstraction or any similarly deep concept. I think that if you had quite a lot of questions (say 20-50 of them) you might be able to get a handle on it, <strong>but it would take very careful cross evaluation of the answers given to be able to say with any confidence</strong>. Getting a single question wrong likely tells you almost nothing. And that, of course, makes MCQ less valuable for its main use - quick and easy evaluation. </p>
  
  <p>I could, on the other hand create a bunch of questions that start to get at the student's understanding. The ones I'd like to use are short response questions, but some can be simple choices. </p>
  
  <p>Assuming that the student have a common language background, something like "Is java.util.List abstract or concrete?" "Is java.util.Collection more or less abstract than java.util.Map". One choice for the answers needs to be <em>neither</em> of course. </p>
  
  <p>You can show code fragments and ask whether they <em>implement an abstraction properly</em> or not. But a follow up "why or why not" gives you better information. And some of those questions would necessarily be a bit long to be meaningful and thus take some time for the student to evaluate. </p>
  
  <p>"How can functions be considered abstractions?" Well, that was poorly stated, since a single function is concrete, not abstract, so you need to be careful: "How can the idea of a function be considered an abstraction?" is better. You could probably give a few suggestions for answers to this to turn it in to a MCQ, but it would be hard to come up with four, of which three were viable and yet not correct. </p>
  
  <p>However, I've only hit the surface here and only explored a very small part of the <em>abstraction space</em> in computing. </p>
  
  <p>Caveat: I never had to teach or evaluate a hundred or more students at a go. My classes were always small enough that I could use better evaluation techniques than MCQ, though would be likely to use a few in a larger assessment that allowed the students more creativity. If you <em>do</em> have to evaluate 800 students I hope you have help. If you don't something is deeply wrong. I note that Harvard CS 50 is run by a team of about 80 people, with quite a number of roles. I hope that each student has some individual person responsible for them. It isn't the professor. </p>
  
  <p>My initial "just say no" is to the idea that you can completely evaluate a student with MCQ, not that they can't be useful, dangerous drug though they are. And note that I have the highest respect for Simon Peyton-Jones. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-06T18:33:17.487Z
Answers.LastActivityDate: 2017-10-06T18:33:17.487Z
Answers.UserDisplayName: Buffy
Answers.Body: <p>Fundamentally, I question the notion that you can test to see if students understand the idea of abstraction beyond a superficial level, even if you don't restrict yourself to asking just multiple choice questions. It's a little hard for me to articulate, but I sort of feel the best way to learn + get feedback on whether you understand abstraction is to actually try tackling a large, novel-ish problem and discuss your design decisions with a more experienced dev (or, if self-teaching, learn the hard way why some decision was a bad idea later down the line).</p>
  
  <p>Nevertheless, here are some ideas (with the caveat that I have never once designed and administered a multiple-choice question style thing which means it's entirely possible I have no idea what I'm talking about).</p>
  
  <p>These are all more CS-y and math-y then your example questions, but I figured you might as well address the skills you're after in a directly applicable setting instead of beating around the bush.</p>
  
  <ol>
  <li><p>Test to see if students understand what <em>implementation detail</em> is -- that is, do the recognize when something "breaks" an abstraction?</p>
  
  <p>For example, you could give students a description of a car, and ask them to identify from a list which element is implementation detail. (For example, "hitting the brakes causes a moving car to slow down or stop" is part of the interface; "a combustion engine powers the car" or "the car moves via wheels" would both be implementation detail).</p>
  
  <p>If you want something more directly related to code, give them a header comment for some method or class, and ask them to do the same.</p></li>
  <li><p>Test to see if students understand how to find similarities and find a viable abstraction between <em>multiple</em> items. (After all, if you have only one of something, why would you bother abstracting it?)</p>
  
  <p>For example, you could give students a description of several different queue-like objects (a FIFO queue, a priority queue, etc) and ask them which interface description would work for all of them. (This probably isn't the best example, but hopefully you get the idea).</p>
  
  <p>I would also add an option that says "none of these are good abstractions" and make that the right answer some of the time. After all, if students are taught that they should always be abstracting, then you'll end up with a bunch of architecture astronauts, which seems suboptimal.</p></li>
  <li><p>Draw inspiration from <a href="https://bartoszmilewski.com/2014/10/28/category-theory-for-programmers-the-preface/" rel="nofollow noreferrer">category theory</a>. </p>
  
  <p>After all, there's a <a href="http://blog.ploeh.dk/2017/10/04/from-design-patterns-to-category-theory/" rel="nofollow noreferrer">argument to be made</a> that constructs from category theory and design patterns are more or less the same thing (except that category theory is applicable to things beyond code and design patterns are more handwavy).</p>
  
  <p>For example, you could perhaps describe something like monoid (e.g. if the set M is a monoid, then it must be the case that that the set is closed under some associative binary operation and there must exist some identity element I in the set such that A op I = A and I op A = A).</p>
  
  <p>You could then explain how strings + string concatenation form a monoid as a case example (the empty string is the identity; concatenation is the associative binary operation), then ask them to correctly identify how lists or numbers or something are a monoid. You could also ask them which entry from a list <em>wouldn't</em> count as a monoid.</p>
  
  <p>This basically lets you test if students can understand how to use some (relatively basic) abstraction/adapt existing code to work with unfamiliar abstractions. (And tests to see if students know how to adapt to dealing with new definitions and such).</p></li>
  <li><p>Give students a snippet of code that breaks an abstraction, and ask them to identify the correct reason why the code is wrong (and probably add an option for "no, everything's fine" to make it a little harder.)</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-06T16:44:59.010Z
Answers.LastActivityDate: 2017-10-08T15:38:38.030Z
Answers.LastEditDate: 2017-10-08T15:38:38.030Z
Answers.UserDisplayName: Michael0x2a
Answers.Body: <p>Your tennis racket question, is not good. As tells as that is is abstraction, then focuses on the form, when the most important aspect of a tennis racket is its function. Therefore the answer is non of the above (those are all implementation detail, and you asked for an abstraction). A good abstraction is hitty thing.</p>
  <p>The 2nd question, is very long, and can be summarised as, “guess the word I am thinking of”. While all answers are plausible, non are obviously wrong. Unless I <code>grep</code> my notes and see that “Thinking Abstractly” is the only one of these phrases that the teacher has used.</p>
  <h1>This is my attempt at an abstraction question.</h1>
  <p>Based on what I learnt on page 24 of <a href="https://helloworld.raspberrypi.org/issues/3" rel="nofollow noreferrer">Hello, World issue 3</a>.</p>
  <blockquote>
  <p>Which of these is an abstraction of a tennis racket?</p>
  <ul>
  <li>A hand held device for hitting a ball with. [correct answer]</li>
  <li>A colander. [A simile: they both have holes in, but do pupils know what a colander is?]</li>
  <li>A handle connected to a frame, with string tied across it, to form a surface for the ball to bounce off of. [An attempt at a detailed description, that is much longer than the other answers.]</li>
  <li>A tennis ball. [ a peer object ]</li>
  </ul>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-05T18:04:28.647Z
Answers.Text: I got my copy of [Hello, World](https://helloworld.raspberrypi.org) the other day. I defiantly recommend it. @Miles's picture is on page 3. You can also read them all on-line.
Answers.UserDisplayName: ctrl-alt-delor
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-05T20:29:49.720Z
Answers.Text: A bat is not an abstraction of a tennis racket.  Bats and tennis rackets are two different specializations of the same abstraction.  The common abstraction is a thing that you can swing with your arms to send a flying ball back in the direction that it came from.  Bats and tennis rackets both have concrete properties that are unlike each other, but which can be "abstracted away" when we want to talk about what they have in common.
Answers.UserDisplayName: Solomon Slow
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-06T07:41:54.783Z
Answers.Text: @jameslarge I think you are correct, can you think of a better word/ phrase?
Answers.UserDisplayName: ctrl-alt-delor
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-06T09:58:17.660Z
Answers.Text: I can see the simile with a colander as something round with holes in, but the closest thing that a tennis racquet seems to have in common with a flying mouse is that both are important scene-setting elements for a genre of books (tennis racquet: story set in upper class English society of the early 20th century; bat: vampire stories). The correct answer to your question is clearly the third one, which abstracts out the essence of a tennis racquet to a definition which also covers badminton racquets, squash racquets, ...
Answers.UserDisplayName: Peter Taylor
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-06T11:38:13.873Z
Answers.Text: Improved options, a bit. If any one has any ideas for getting them all about the some length, and similar grammar structure, then please improve them.
Answers.UserDisplayName: ctrl-alt-delor
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-05T18:01:53.750Z
Answers.LastActivityDate: 2017-10-06T11:37:10.743Z
Answers.LastEditDate: 2020-06-18T08:36:57.023Z
Answers.UserDisplayName: ctrl-alt-delor
Answers.Body: <p>I would probably ask students to demonstrate their understanding of abstraction by writing code demonstrating solutions to a small problem at, say, three different levels of abstraction. I'd probably leave it to them to choose the levels of abstraction, and optionally write a short explanation of the differences in level of abstraction the code was intended to demonstrate.</p>
  
  <p>I think multiple choice questions are highly effective for measuring awareness of facts. I see them as much less effective for determining understanding of ideas.</p>
  
  <p>Looking at the specific questions you've posted, I think both of them are relatively poor. For example, when looking at a tennis racket, I suppose your intent was probably that "rectangle and circle" would be the correct answer. However, that's looking only at the outline of the racket. To a tennis player, the two parts that matter are the strings (which form squares) and the ball that it hits (a circle). So, rectangle and circle might be a good choice if you're thinking primarily in terms of computer graphics, but square and circle might be a better choice if you think in terms of a tennis racket's actual purpose.</p>
  
  <p>Shapes, aren't really much of an abstraction. Shapes deal purely with the physical embodiment of the device. To test for abstract thinking, you need to start by thinking abstractly yourself.</p>
  
  <p>If I were going to try to do this with multiple-choice questions, I'd probably have two in a row that contrast concrete and abstract thinking:</p>
  
  <p>On a physical level (the shape of the concrete implementation), which of these most closely resembles a tennis racket?</p>
  
  <ol>
  <li>Golf club</li>
  <li>lollipop</li>
  <li>automobile</li>
  <li>tennis ball</li>
  </ol>
  
  <p>On an abstract level, which of these most closely resembles a tennis racket?</p>
  
  <ol>
  <li>Golf club</li>
  <li>lollipop</li>
  <li>automobile</li>
  <li>tennis ball</li>
  </ol>
  
  <p>In the first case, the correct answer is '2'. A tennis racket is shaped like a large lollipop.</p>
  
  <p>On a more abstract level, a tennis racket is a tool for hitting a ball. So, at the more abstract level, the most similar is a golf club--also a tool for hitting a ball.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-05T20:58:39.380Z
Answers.Text: The two questions posted are there to show that mcqs for abstraction are generally poor. Can we write good ones? You've made an attempt above,  it would be good to get a clear one answer question with valid distractors
Answers.UserDisplayName: pluke
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-05T18:05:48.723Z
Answers.LastActivityDate: 2017-10-05T18:05:48.723Z
Answers.UserDisplayName: Jerry Coffin
Body: <p>We've just uploaded <a href="https://www.youtube.com/watch?v=M2Zerjl4Vmo" rel="nofollow noreferrer">a video of a presentation</a> Simon Peyton Jones and I did about <a href="http://bit.ly/projectquantum" rel="nofollow noreferrer">Project Quantum</a> to CAS TV. Project Quantum is an attempt to crowd-source low-stakes formative assessment items for computing; we've just over 5,500 questions at the moment.</p>
  
  <p>I boldly (and perhaps erroneously) claim that there's no bit of the computing curriculum for which we can't create multiple choice questions (MCQs). Note, this is rather different from claiming we only need to use MCQs to assess learning in computing, which is very far from what I believe.</p>
  
  <p>My colleague Pete Kemp remarks on <a href="https://www.facebook.com/mberry001/posts/1580631735308765" rel="nofollow noreferrer">the Facebook</a> that he's "still waiting on a Q for abstraction that isn't about definitions...". <a href="https://diagnosticquestions.com/Questions?CurrentSubjectId=1688&amp;OrderBy=Newest&amp;IsByStudent=False" rel="nofollow noreferrer">Here</a>'s what we have on the Diagnostic Questions site at present (free registration required).</p>
  
  <p>So how do <em>you</em> assess whether your pupils understand, or indeed can apply, abstraction? Could you knit this into an MCQ?</p>
  
  <p>Examples (of far from satisfactory questions) from <a href="https://diagnosticquestions.com/Questions?CurrentSubjectId=1688&amp;OrderBy=Newest&amp;IsByStudent=False" rel="nofollow noreferrer">https://diagnosticquestions.com/Questions?CurrentSubjectId=1688&amp;OrderBy=Newest&amp;IsByStudent=False</a> All CC-SA
  <a href="https://i.sstatic.net/6oJcN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6oJcN.png" alt="subjective  MCQ"></a></p>
  
  <p><a href="https://i.sstatic.net/AiCBF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AiCBF.png" alt="tests a defintion MCQ"></a></p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-05T10:41:47.923Z
Comments.Text: For example, a bad question on DQs: Which of the following shapes is an abstraction of the moon: Rectangle | Cuboid | Sphere | Triangle. It's very subjective, e.g. http://www.askamathematician.com/wp-content/uploads/2011/05/cubeearth.jpg
Comments.UserDisplayName: pluke
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-05T10:43:14.133Z
Comments.Text: What is MCQ? Also can you put all important info into the question. Not (just) link, especially to subscription sites.
Comments.UserDisplayName: ctrl-alt-delor
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-05T16:40:37.947Z
Comments.Text: I would struggle with getting those questions correct.
Comments.UserDisplayName: ctrl-alt-delor
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-05T16:45:15.180Z
Comments.Text: A problem with MCQ and abstractions is that often an abstraction has personal context. I also find those examples hard (the first one reads to me as {blah} {blah} for the first 3 choices, and misdirects me because I want to make them all fit). Too much 3D printing, everything decomposes to triangles...
Comments.UserDisplayName: Sean Houlihane
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-05T16:51:29.883Z
Comments.Text: A tennis racket is a white **rectangle**, that moves vertically, and is controlled with a rotary knob.
Comments.UserDisplayName: ctrl-alt-delor
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-05T16:57:02.017Z
Comments.Text: I just looked at the site, there are some flow charts. They made me say wt*?  There is a diamond with the word payment in it. Even with an up inflection that is not a question. If I was doing this quiz I would just answer A for everything or walk out.
Comments.UserDisplayName: ctrl-alt-delor
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-05T17:10:24.187Z
Comments.Text: Am watching the video and am impressed with what I hear. I was thinking that there is a need for up/down votes for questions. However the approach you are taking of the quiz also marking it self, seems better. I wonder if it would be of value to do both. To compare, or as an initial score for the question (would it speed up the systems initial learning). Could you use the system to grade teacher/assessors ability to assess questions.
Comments.UserDisplayName: ctrl-alt-delor
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-06T09:05:28.243Z
Comments.Text: As a programmer,  not a teacher,  the question looks to me to be presented as recipe regurgitation. I would never approach the font problem in the manner the question presents it. A person understands abstraction when they can look at a specific problem and grasp that there is a larger set of problems that the initial problem is a mere part of, which can be resolved with the same mechanisms. Recognizing that the existence of a font implies multiple fonts and the need for a font manager to hide the font implementation from the core business of display is abstraction.
Comments.UserDisplayName: pojo-guy
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-06T11:47:49.503Z
Comments.Text: Users can 'favourite' questions and include them in quizzes - we can sort the questions on these metrics. Users can also provide feedback to question writers. 
  The Rasch analysis will be online "later this year", which will provide a measure of how hard a question is and how good it is at discriminating strong and weak students. Users and writers will have access to this data too.
Comments.UserDisplayName: Miles
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-06T17:10:03.807Z
Comments.Text: "Understanding of Abstraction" is not something that can readily be measured... (Wittgenstein got it wrong too)  And the constraints due to nature of MCQs probably requires a group of questions rather than a single one. Probably psychology, psychiatry, social studies, anthropology and industrial-psychology are good disciplines to tap as they have been grappling with similar problems for a lot longer. A CS'esq approach: Derive questions out of a graph of concepts and apply a (biased) distance-function between the query node and the answer nodes, with bias declared in the query.
Comments.UserDisplayName: Ahmed Masud
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-09T12:54:37.573Z
Comments.Text: Aren't questions that ask the student to group objects or find an object not belonging to a group already testing abstract thinking? In the end of the day, this is what it is: you get a bunch of objects, imagine an abstract concept that might fit all of them or most of them--and here you are, making abstractions.
Comments.UserDisplayName: wvxvw
Tags: <assessment><abstraction><computational-thinking>
Title: How do you assess students' understanding of abstraction?
ThreadText: # How do you assess students' understanding of abstraction?
  
  (Asked by: username_0 on 2017-10-05)
  
  We've just uploaded [a video of a presentation](https://www.youtube.com/watch?v=M2Zerjl4Vmo) Simon Peyton Jones and I did about [Project Quantum](http://bit.ly/projectquantum) to CAS TV. Project Quantum is an attempt to crowd-source low-stakes formative assessment items for computing; we've just over 5,500 questions at the moment.
  
  I boldly (and perhaps erroneously) claim that there's no bit of the computing curriculum for which we can't create multiple choice questions (MCQs). Note, this is rather different from claiming we only need to use MCQs to assess learning in computing, which is very far from what I believe.
  
  My colleague Pete Kemp remarks on [the Facebook](https://www.facebook.com/mberry001/posts/1580631735308765) that he's "still waiting on a Q for abstraction that isn't about definitions...". [Here](https://diagnosticquestions.com/Questions?CurrentSubjectId=1688&OrderBy=Newest&IsByStudent=False)'s what we have on the Diagnostic Questions site at present (free registration required).
  
  So how do *you* assess whether your pupils understand, or indeed can apply, abstraction? Could you knit this into an MCQ?
  
  Examples (of far from satisfactory questions) from <https://diagnosticquestions.com/Questions?CurrentSubjectId=1688&OrderBy=Newest&IsByStudent=False> All CC-SA
  [![subjective  MCQ](https://i.sstatic.net/6oJcN.png)](https://i.sstatic.net/6oJcN.png)
  
  [![tests a defintion MCQ](https://i.sstatic.net/AiCBF.png)](https://i.sstatic.net/AiCBF.png)
  
  **Tags:** assessment, abstraction, computational-thinking
  
  > **Comments:**
  > **username_1**: For example, a bad question on DQs: Which of the following shapes is an abstraction of the moon: Rectangle | Cuboid | Sphere | Triangle. It's very subjective, e.g. http://www.askamathematician.com/wp-content/uploads/2011/05/cubeearth.jpg  
  > **username_2**: What is MCQ? Also can you put all important info into the question. Not (just) link, especially to subscription sites.  
  > **username_2**: I would struggle with getting those questions correct.  
  > **username_3**: A problem with MCQ and abstractions is that often an abstraction has personal context. I also find those examples hard (the first one reads to me as {blah} {blah} for the first 3 choices, and misdirects me because I want to make them all fit). Too much 3D printing, everything decomposes to triangles...  
  > **username_2**: A tennis racket is a white **rectangle**, that moves vertically, and is controlled with a rotary knob.  
  > **username_2**: I just looked at the site, there are some flow charts. They made me say wt*?  There is a diamond with the word payment in it. Even with an up inflection that is not a question. If I was doing this quiz I would just answer A for everything or walk out.  
  > **username_2**: Am watching the video and am impressed with what I hear. I was thinking that there is a need for up/down votes for questions. However the approach you are taking of the quiz also marking it self, seems better. I wonder if it would be of value to do both. To compare, or as an initial score for the question (would it speed up the systems initial learning). Could you use the system to grade teacher/assessors ability to assess questions.  
  > **username_4**: As a programmer,  not a teacher,  the question looks to me to be presented as recipe regurgitation. I would never approach the font problem in the manner the question presents it. A person understands abstraction when they can look at a specific problem and grasp that there is a larger set of problems that the initial problem is a mere part of, which can be resolved with the same mechanisms. Recognizing that the existence of a font implies multiple fonts and the need for a font manager to hide the font implementation from the core business of display is abstraction.  
  > **username_0**: Users can 'favourite' questions and include them in quizzes - we can sort the questions on these metrics. Users can also provide feedback to question writers.  The Rasch analysis will be online "later this year", which will provide a measure of how hard a question is and how good it is at discriminating strong and weak students. Users and writers will have access to this data too.  
  > **username_5**: "Understanding of Abstraction" is not something that can readily be measured... (Wittgenstein got it wrong too)  And the constraints due to nature of MCQs probably requires a group of questions rather than a single one. Probably psychology, psychiatry, social studies, anthropology and industrial-psychology are good disciplines to tap as they have been grappling with similar problems for a lot longer. A CS'esq approach: Derive questions out of a graph of concepts and apply a (biased) distance-function between the query node and the answer nodes, with bias declared in the query.  
  > **username_6**: Aren't questions that ask the student to group objects or find an object not belonging to a group already testing abstract thinking? In the end of the day, this is what it is: you get a bunch of objects, imagine an abstract concept that might fit all of them or most of them--and here you are, making abstractions.  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_8 on 2017-10-06. Score: 7
  
  tl;dr: Just say no.
  
  This question is difficult on many levels. I seems to me to be a land mine of misconceptions and has the possibility to lead to poor teaching practice.
  
  First the difficulties
  
  1. Only *concepts* can be abstract. Abstraction is about ideas. Animal is a concept. Mammal is a concept. Animal is more abstract than Mammal since it contains the idea of Mammal, but isn't restricted to it. The kitty sitting on my hearth is concrete, not less abstract than Mammal. The kitty isn't a concept it is a thing.
  2. Given some ot the examples, there is a misconception about what is abstract and what is concrete. Physical things are not abstract. One can't be *more* abstract than another. Even things that aren't physical, but which can be manipulated, are concrete, not abstract. A LinkedList (Java) isn't abstract. It is concrete. List, on the other hand (an interface) is abstract.
  3. In computing there are many, many different kinds of abstraction. Even the concept of a while loop is an abstraction. In OOP languages classes implement abstractions. Interfaces (even if the language doesn't have that explicit feature) are abstractions. So, Python has interfaces, though it has no language feature with that name. If two classes have the same interface they are realizations of the abstraction, but neither is more or less abstract than the other. If one interface contains another interface then it is more abstract.But we also have *data abstraction*. Lists and Maps define an interface (an abstraction). They can be implemented/realized many ways. Each realization is concrete, not less abstract.
  4. The web site cited seems to be trying to test everything with Multiple Choice Questions (MCQ). I question both the possibility of that and the wisdom of it. Some thing are complex MCQ are simple. Trying to push the assessment of complex things through such a small funnel seems misplaced to me - even foolish. I'm pretty sure that MCQ also advantage certain students and disadvantage others. They are fine for rote memory and facts, but pretty poor for deep understanding. I have nothing against their use unless it is overdone. MCQ are also difficult to validate. The wisdom of groups isn't enough. It takes statistical measurement and cross validation to do a good job of it.
  5. The Font Construction example given in the question also seem to be testing specific vocabulary. This example threw me completely. I think I could eliminate one answer, but also think I could justify each of the others if given the chance. If it is specific vocabulary it is just rote memory and hoping that the student "remembered" to remember the right thing.
  6. Often, getting a MCQ wrong and then later learning the *correct* answer can sometimes leave students simply puzzled and frustrated. This results in a destructive game that I'd rather not play.
  7. MCQ try to put a lot of thinking into the questions so that no one needs to put any thinking into the answers after the student finishes. This may scale well, but is a sub-optimal form of *education*. After all, it is the *students thoughts* that we want to get a handle on in assessment.
  
  I don't believe that a single, or even a small number of MCQ could capture the student's understanding of abstraction or any similarly deep concept. I think that if you had quite a lot of questions (say 20-50 of them) you might be able to get a handle on it, **but it would take very careful cross evaluation of the answers given to be able to say with any confidence**. Getting a single question wrong likely tells you almost nothing. And that, of course, makes MCQ less valuable for its main use - quick and easy evaluation.
  
  I could, on the other hand create a bunch of questions that start to get at the student's understanding. The ones I'd like to use are short response questions, but some can be simple choices.
  
  Assuming that the student have a common language background, something like "Is java.util.List abstract or concrete?" "Is java.util.Collection more or less abstract than java.util.Map". One choice for the answers needs to be *neither* of course.
  
  You can show code fragments and ask whether they *implement an abstraction properly* or not. But a follow up "why or why not" gives you better information. And some of those questions would necessarily be a bit long to be meaningful and thus take some time for the student to evaluate.
  
  "How can functions be considered abstractions?" Well, that was poorly stated, since a single function is concrete, not abstract, so you need to be careful: "How can the idea of a function be considered an abstraction?" is better. You could probably give a few suggestions for answers to this to turn it in to a MCQ, but it would be hard to come up with four, of which three were viable and yet not correct.
  
  However, I've only hit the surface here and only explored a very small part of the *abstraction space* in computing.
  
  Caveat: I never had to teach or evaluate a hundred or more students at a go. My classes were always small enough that I could use better evaluation techniques than MCQ, though would be likely to use a few in a larger assessment that allowed the students more creativity. If you *do* have to evaluate 800 students I hope you have help. If you don't something is deeply wrong. I note that Harvard CS 50 is run by a team of about 80 people, with quite a number of roles. I hope that each student has some individual person responsible for them. It isn't the professor.
  
  My initial "just say no" is to the idea that you can completely evaluate a student with MCQ, not that they can't be useful, dangerous drug though they are. And note that I have the highest respect for Simon Peyton-Jones.
  
  
  ---
  
  ### Answer by username_9 on 2017-10-06. Score: 4
  
  Fundamentally, I question the notion that you can test to see if students understand the idea of abstraction beyond a superficial level, even if you don't restrict yourself to asking just multiple choice questions. It's a little hard for me to articulate, but I sort of feel the best way to learn + get feedback on whether you understand abstraction is to actually try tackling a large, novel-ish problem and discuss your design decisions with a more experienced dev (or, if self-teaching, learn the hard way why some decision was a bad idea later down the line).
  
  Nevertheless, here are some ideas (with the caveat that I have never once designed and administered a multiple-choice question style thing which means it's entirely possible I have no idea what I'm talking about).
  
  These are all more CS-y and math-y then your example questions, but I figured you might as well address the skills you're after in a directly applicable setting instead of beating around the bush.
  
  1. Test to see if students understand what *implementation detail* is -- that is, do the recognize when something "breaks" an abstraction?
  
     For example, you could give students a description of a car, and ask them to identify from a list which element is implementation detail. (For example, "hitting the brakes causes a moving car to username_10 down or stop" is part of the interface; "a combustion engine powers the car" or "the car moves via wheels" would both be implementation detail).
  
     If you want something more directly related to code, give them a header comment for some method or class, and ask them to do the same.
  2. Test to see if students understand how to find similarities and find a viable abstraction between *multiple* items. (After all, if you have only one of something, why would you bother abstracting it?)
  
     For example, you could give students a description of several different queue-like objects (a FIFO queue, a priority queue, etc) and ask them which interface description would work for all of them. (This probably isn't the best example, but hopefully you get the idea).
  
     I would also add an option that says "none of these are good abstractions" and make that the right answer some of the time. After all, if students are taught that they should always be abstracting, then you'll end up with a bunch of architecture astronauts, which seems suboptimal.
  3. Draw inspiration from [category theory](https://bartoszmilewski.com/2014/10/28/category-theory-for-programmers-the-preface/).
  
     After all, there's a [argument to be made](http://blog.ploeh.dk/2017/10/04/from-design-patterns-to-category-theory/) that constructs from category theory and design patterns are more or less the same thing (except that category theory is applicable to things beyond code and design patterns are more handwavy).
  
     For example, you could perhaps describe something like monoid (e.g. if the set M is a monoid, then it must be the case that that the set is closed under some associative binary operation and there must exist some identity element I in the set such that A op I = A and I op A = A).
  
     You could then explain how strings + string concatenation form a monoid as a case example (the empty string is the identity; concatenation is the associative binary operation), then ask them to correctly identify how lists or numbers or something are a monoid. You could also ask them which entry from a list *wouldn't* count as a monoid.
  
     This basically lets you test if students can understand how to use some (relatively basic) abstraction/adapt existing code to work with unfamiliar abstractions. (And tests to see if students know how to adapt to dealing with new definitions and such).
  4. Give students a snippet of code that breaks an abstraction, and ask them to identify the correct reason why the code is wrong (and probably add an option for "no, everything's fine" to make it a little harder.)
  
  
  ---
  
  ### Answer by username_12 on 2017-10-05. Score: 4
  
  I would probably ask students to demonstrate their understanding of abstraction by writing code demonstrating solutions to a small problem at, say, three different levels of abstraction. I'd probably leave it to them to choose the levels of abstraction, and optionally write a short explanation of the differences in level of abstraction the code was intended to demonstrate.
  
  I think multiple choice questions are highly effective for measuring awareness of facts. I see them as much less effective for determining understanding of ideas.
  
  Looking at the specific questions you've posted, I think both of them are relatively poor. For example, when looking at a tennis racket, I suppose your intent was probably that "rectangle and circle" would be the correct answer. However, that's looking only at the outline of the racket. To a tennis player, the two parts that matter are the strings (which form squares) and the ball that it hits (a circle). So, rectangle and circle might be a good choice if you're thinking primarily in terms of computer graphics, but square and circle might be a better choice if you think in terms of a tennis racket's actual purpose.
  
  Shapes, aren't really much of an abstraction. Shapes deal purely with the physical embodiment of the device. To test for abstract thinking, you need to start by thinking abstractly yourself.
  
  If I were going to try to do this with multiple-choice questions, I'd probably have two in a row that contrast concrete and abstract thinking:
  
  On a physical level (the shape of the concrete implementation), which of these most closely resembles a tennis racket?
  
  1. Golf club
  2. lollipop
  3. automobile
  4. tennis ball
  
  On an abstract level, which of these most closely resembles a tennis racket?
  
  1. Golf club
  2. lollipop
  3. automobile
  4. tennis ball
  
  In the first case, the correct answer is '2'. A tennis racket is shaped like a large lollipop.
  
  On a more abstract level, a tennis racket is a tool for hitting a ball. So, at the more abstract level, the most similar is a golf club--also a tool for hitting a ball.
  
  > **Comments:**
  > **username_1**: The two questions posted are there to show that mcqs for abstraction are generally poor. Can we write good ones? You've made an attempt above,  it would be good to get a clear one answer question with valid distractors  
  
  ---
  
  ### Answer by username_2 on 2017-10-05. Score: 3
  
  Your tennis racket question, is not good. As tells as that is is abstraction, then focuses on the form, when the most important aspect of a tennis racket is its function. Therefore the answer is non of the above (those are all implementation detail, and you asked for an abstraction). A good abstraction is hitty thing.
  
  The 2nd question, is very long, and can be summarised as, “guess the word I am thinking of”. While all answers are plausible, non are obviously wrong. Unless I `grep` my notes and see that “Thinking Abstractly” is the only one of these phrases that the teacher has used.
  
  # This is my attempt at an abstraction question.
  
  Based on what I learnt on page 24 of [Hello, World issue 3](https://helloworld.raspberrypi.org/issues/3).
  
  > Which of these is an abstraction of a tennis racket?
  >
  > * A hand held device for hitting a ball with. [correct answer]
  > * A colander. [A simile: they both have holes in, but do pupils know what a colander is?]
  > * A handle connected to a frame, with string tied across it, to form a surface for the ball to bounce off of. [An attempt at a detailed description, that is much longer than the other answers.]
  > * A tennis ball. [ a peer object ]
  
  > **Comments:**
  > **username_2**: I got my copy of [Hello, World](https://helloworld.raspberrypi.org) the other day. I defiantly recommend it. @username_0's picture is on page 3. You can also read them all on-line.  
  > **username_10**: A bat is not an abstraction of a tennis racket.  Bats and tennis rackets are two different specializations of the same abstraction.  The common abstraction is a thing that you can swing with your arms to send a flying ball back in the direction that it came from.  Bats and tennis rackets both have concrete properties that are unlike each other, but which can be "abstracted away" when we want to talk about what they have in common.  
  > **username_2**: @jameslarge I think you are correct, can you think of a better word/ phrase?  
  > **username_11**: I can see the simile with a colander as something round with holes in, but the closest thing that a tennis racquet seems to have in common with a flying mouse is that both are important scene-setting elements for a genre of books (tennis racquet: story set in upper class English society of the early 20th century; bat: vampire stories). The correct answer to your question is clearly the third one, which abstracts out the essence of a tennis racquet to a definition which also covers badminton racquets, squash racquets, ...  
  > **username_2**: Improved options, a bit. If any one has any ideas for getting them all about the some length, and similar grammar structure, then please improve them.  
  
  ---
  
  ### Answer by username_7 on 2017-10-06. Score: 0
  
  The best way to find out is to give programming problems, or similarly complex design problems, where you can look at the details of the solutions they create. I give lots of "lab" assignments where the students have to write programs based on a problem statement from the textbook. Part of the answer is: "Are they using what we just covered in class?", but the vital part is: **"Are they gaining knowledge over time and learning to assemble things with increasingly complex structure?"**
  
  Some students clearly learn general concepts like how to define classes and use them sensibly, while other students struggle with this, or even more basic concepts like how to use variables in one UI method that were set in another UI method. (Could be Console or Windows, doesn't matter.) Some students take the ideas of Relational Comparison and Boolean combiners and use them correctly, others struggle and try to create things by sticking together the code they painfully got to work in the previous assignment (which they are now trying to re-use for an opposite case).
  
  In short, *assessing* whether students have learned how to abstract is bone-obvious. *Teaching* it is the hard part. **We need to find ways to teach abstraction.** My research has turned up very little except arguments over what the word means, or worse: whether to start teaching at high levels of abstraction, or instead teach how computers actually work, etc. (See other questions on this site.) Poor start on a very significant problem!
  
  > **Comments:**
  > **username_8**: Welcome to CSEducators. Hope to see you again. I'm one who isn't so sure that the assessment is obvious, but agree that teaching is the hard part.  
  
  ---
  
  