The Lesson to Unlearn
As somebody who has designed some of the tests Paul Graham is complaining about, it really is hard from the other side!<p>For example, it&#x27;s long been known in the physics education research community that students come away from introductory courses with very little physical understanding, even if they can do the plug and chug problems on typical tests just fine. Students can all recite Newton&#x27;s third law, but immediately afterward claim that when a truck hits a car, the truck exerts a bigger force. They know the law for the gravitational force, but can&#x27;t explain what kept astronauts from falling off the moon, since &quot;there&#x27;s no gravity in space&quot;. Another common claim is that a table exerts no force on something sitting on it -- instead of &quot;exerting a force&quot; it&#x27;s just &quot;getting in the way&quot;.<p>For research purposes, we measure physical understanding using a battery of tests, such as the Force Concept Inventory, containing only simple conceptual questions with unambiguous answers. So then everybody asks: if ordinary tests are so hackable, why not just switch to these conceptual ones? But that wouldn&#x27;t work. There are less than ~100 distinct FCI-style questions. If these conceptual tests were the norm, students would just memorize the answers and parrot them back, with a flimsy understanding that crumples the second any follow-up question is asked. It would be just the same problem as before, except they would be worse computationally, too. The FCI only works as long as it doesn&#x27;t count for a grade.<p>The problem isn&#x27;t tests, it&#x27;s scale. If the people aren&#x27;t motivated, any standardized measure will miss the mark -- even the entrepreneurship Paul Graham advocates for. God knows I&#x27;ve seen a lot of bullshit in that direction.