How not to be a crank: rules for not being a science-dick
I&#x27;ve also investigated bad science papers before, and blogged about it a few times. There&#x27;s a lot out there. There&#x27;s nothing special about getting a PhD that makes you politically neutral or morally superior. The temptation to abuse maths to make your personal views seem &quot;scientific&quot; is irresistible for too many. My last public attempt:<p><a href="https:&#x2F;&#x2F;blog.plan99.net&#x2F;did-russian-bots-impact-brexit-ad66f08c014a" rel="nofollow">https:&#x2F;&#x2F;blog.plan99.net&#x2F;did-russian-bots-impact-brexit-ad66f...</a><p>I feel some sympathy with fellow Brit Nick Brown after reading this story about his work:<p><a href="https:&#x2F;&#x2F;www.theguardian.com&#x2F;science&#x2F;2014&#x2F;jan&#x2F;19&#x2F;mathematics-of-happiness-debunked-nick-brown" rel="nofollow">https:&#x2F;&#x2F;www.theguardian.com&#x2F;science&#x2F;2014&#x2F;jan&#x2F;19&#x2F;mathematics-...</a><p>(as an aside, totally unsurprised the guy works in computing, developing an intolerance for abuse of logic is a job hazard.)<p>But I don&#x27;t know if this perspective is helpful to be honest. &quot;Don&#x27;t be a dick&quot; is both quite an obvious principle and also rather useless when criticising people&#x27;s work: some of the authors, and some of the people who want to believe the research is true, will inevitably view any criticism as being dickish. This is doubly true if you take the obvious next step and speculate as to whether the issues are mere oversights or (far more commonly, in my view) deliberate deceptions intended to further an agenda.<p>One problem is that genuine, honest mistakes tend to get picked up already by the existing peer review process. The really bad papers that get through are usually bad in support of some wider social mission, usually leading the public to some policy goal, and they don&#x27;t get struck down because the peer reviewers share the same goals. Thus pointing out mistakes has no effect, because they already knew the science was bad. The only thing that can work is pointing out to the people who they&#x27;re trying to influence that something has gone wrong.<p>There <i>is</i> a responsibility to be harsh in these cases. Failing to do so can simply let the issues fester and compound. From the Guardian article on Brown:<p><i>There were several psychologists, versed in non-linear dynamics, who smelt something fishy about the maths in the published paper. Stephen Guastello, from Marquette University, wrote a note of mild complaint to American Psychologist, which it chose not to publish because &quot;there wasn&#x27;t enough interest in the article&quot;. Guastello feels now that he should have been more forceful in his opinions. &quot;In retrospect,&quot; he says, &quot;I see how I could have been more clearly negative and less supportive of what looked like an article that could move the field forward if someone would follow up with some strong empirical work.&quot;</i><p>The story describes Brown&#x27;s first debunking of a paper that used lots of clever looking math to reach obviously absurd conclusions about psychology. One author of the paper admitted that she had never understood the maths and the person who did create the maths (Losada) has refused to ever respond to the debunking in any way. <i>All</i> the reviewers were successfully intimidated by the maths and were unwilling to criticise it, allowing Losada to get away with it:<p><i>John Gottman, a leading authority in the psychology of successful relationships, wrote to Losada because he couldn&#x27;t follow the equations. &quot;I thought it was something I didn&#x27;t know about, because he&#x27;s a smart guy, Losada. He never answered my email,&quot; he says.</i><p>So the critics weren&#x27;t critical enough and a paper based on pure mathematical bullshit racked up 350 citations. Vast amounts of time, money and effort was wasted. What does this say about the scientific process?<p>By the way, in case anyone thinks computing is immune to this sort of thing, it&#x27;s not. Papers about &quot;Russian twitter bots&quot; are a current streak of bad science in computing, for obvious reasons: the authors want to undermine political trends they don&#x27;t like. The last one I read claimed to have constructed a neural network that could detect Russian bots with 99%+ accuracy. Nowhere in the paper did it give any examples of what the network detected, nor what the ground truth they were comparing to was. When I emailed the authors to ask for the data so the study could be replicated, I was told the data was not available (despite being just a bunch of tweets i.e. public data).<p>It was literally an entire paper that looked scientific but boiled down to &quot;we made a magic black box that without fail finds evidence of a vast conspiracy, but we can&#x27;t show you. Trust us.&quot;