Is novel research worth doing? Evidence from peer review
I think often of this jab by Rodney Brooks on my home field of human-robot interaction [1]:<p>&quot;A group of people who have never used a computer are recruited for an experiment, using variations of Unix commands to manipulate files from a command line interface. The users have never heard of computer files before and they don’t in any sense have the utility of files internalized.<p>The experiment is to determine whether something the experimenters have called utility versus expressiveness is more highly valued. One set of users is taught commands like “rm,” “mkdir,” and “cat,” etc., while another is taught “delete,” “new_directory,” and “type_out,” etc. They are asked to do some housekeeping tasks on a file system. Their speed and accuracy is measured, and they fill in a questionnaire about how hard and how intuitive the task was. The paper itself about these experiments touts “statistically significant results,” with lots of p-values, etc.<p>No HCI academics bother to invent windows, icons, a mouse, and so on. Such work is viewed as too informal and unscientific.&quot;<p>I just finished my PhD and these lines were in my head since they were published. Between an advisor who would only greenlight plans if they immediately led to a paper and an embarrassing amount of impractical &quot;safe&quot; research that only makes for good tweets or headlines, I&#x27;ve given up hope on anything meaningful coming out of the Ivory Towers. The robot revolution will not be peer reviewed.<p>[1] <a href="https:&#x2F;&#x2F;dl.acm.org&#x2F;doi&#x2F;10.1145&#x2F;3209540" rel="nofollow">https:&#x2F;&#x2F;dl.acm.org&#x2F;doi&#x2F;10.1145&#x2F;3209540</a>