Cargo Cult Science (1974) [pdf]
I&#x27;m only really familiar with machine learning and computer vision papers and many of the mentioned aspects, especially Mr. Young&#x27;s rat running experiments have their analogs here.<p>There is a ton of accumulated &quot;dark knowledge&quot; locked up in research groups, all the tricks (like putting the rat corridor on sand, in that example) but they are not really publishable. To publish, you need a clear <i>story</i>, literally that&#x27;s the word we use when drafting papers. What&#x27;s your story? A bunch of boring tricks rarely make a good story. And when people do publish such things, they have to also insert some other conceptual &quot;novelty&quot; contribution (typically a non intuitive tweak of the model architecture) to the paper that makes a +0.5% improvement on some benchmark, just to be able to talk about the real practical but ugly things that really make the whole thing work.<p>Not to mention the replications that Feynman mentions, ie that before you test your tweaked new model, you have to also perform the baseline yourself, you can&#x27;t just take it as given in another person&#x27;s paper. This is rarely done in ML, and not just for resource scarcity reasons. Another reason is that it&#x27;s damn hard. ML systems are very very complex and essentially impossible to exactly reproduce from a paper description. It would even be hard for the same team to do it, if we deleted their codebase and checkpoints and asked them to redo it.<p>&quot;But open source!&quot; you say. Sure, except that large systems often evolve and the released code is often a refactored version of a haphazard ducktaped spaghetti monster codebase that was actually used, where they manually edited code files between runs, or hard-coded things, discovered some Bug midway and fixed it and compensated for it best they could etc.<p>These projects must be done in a few months so youre ready before the next conference cycle happens where someone does something related and now you need to rework your story and contribution claim, or at least you now also have to compare and compete with them.<p>But let&#x27;s say you took your time and now redid the experiment of the other prior work, but it doesn&#x27;t agree with your numbers totally. You can email them or open a github issue. They answer perhaps in a week, and say they don&#x27;t know exactly the reason, or that it was a different code version they actually used but they can&#x27;t release that one as it&#x27;s not approved by corporate (from real experience). Of course with your questions you are also terrifying them and your queries may feel to them like threats of pending potential reputation destruction. So they will be very defensive, which will make you suspicious. But they are just some other grad student like you and probably didn&#x27;t mean any ill.<p>I&#x27;ve seen a PhD student on Twitter asking what he should do after discovering an anomaly in an Arxiv paper before the camera ready deadline (the finalized version of an article). Should he publicly &quot;out&quot; them if they fail to incorporate his findings. Of course it is absurd and the camera ready deadline cannot introduce significant new contributions or new discoveries, as that would require new peer review. But he was convinced that he&#x27;s a noble defender of scientific integrity while doing this. I&#x27;m just mentioning this because some junior people may read these Feynman pieces and think they should go on a crusade based on often quite scarce information.<p>But again, think of the sheer volume of works coming out every week. It&#x27;s unmanageable. If you stopped to interact with every single prior work of your comparison table in such detail you&#x27;d take years to write one paper. A PhD student usually has to publish 3 top papers in about 4 years or so. And some will definitely get rejected. The reality is that high-end labs have become paper factories. They have a process, just like pop songs are formulaic. They get a smart person as an intern for example and pump out a paper in 5 months. Exactly how much of this &quot;you are the easiest person to fool&quot; deep self-reflection fits into such a thing?<p>And yet.<p>And yet the cumulative effect is undeniable progress. The thousands of low quality papers are simply ignored. They contribute to someone getting their PhD, and that&#x27;s their true function. But then there is a small set of works that really are excellent. It&#x27;s just that the publicly available papertrail in the literature isn&#x27;t really necessarily what has driven it. The papers are more like a shadow projection of the real world work behind the scenes, filtered to please novelty-hungry impatient reviewers and paper-count-rewarding committees. But of course the sheer hardware growth is a big part of the overall success, but the hardware design was informed by the research, and without the model improvements, you couldn&#x27;t just hardware-scale the state of the art of 1995 to modern computes and expect strong results.<p>So for sure the spirit that Feynman espouses here still lives on, but it&#x27;s alive despite all the incentives, and most of what appears as academic science is not really about contributing some truly usable and convincing knowledge, but a demonstration of the job skills of people towards various personal evaluations, like granting a degree, hiring or promotion.<p>Most people who start with this starry eyed idealism quickly get it stamped out by the system. The important thing is to yield a productive synthesis instead of a resignatory pessimism. Do your best given the circumstances, but also read the room and don&#x27;t run with your head into the wall.<p>The fabulous thing is though, that things adapt. The less these hurried processes live up to the ideal, the more the reputation of the label &quot;science&quot; gets eroded. Many people already react with an eye roll when they hear what &quot;experts&quot; and &quot;The Science&quot; have to say. The trust is finite and can run out.<p>A few major discoveries in physics and medicine led to a giant reserve of public trust over the last century, but it isn&#x27;t infinite. Immediately after the moon landing, in the space age, science and scifi captivated the minds of everyone and that was probably the peak of it, including figures like Sagan (or indeed Feynman). Then computing brought a new wave of tech but it, and even AI is less of a natural science, and many popular claims turn out to be overblown.<p>---<p>Anyway, we have no idea what exactly made the 100 years between, say, 1870 and 1970 so scientifically productive. Because that&#x27;s the period that lends the weight to the label &quot;science&quot; in the public and hence for politicians. It certainly wasn&#x27;t the current academic system of journals and conferences and 8-page papers and rushed peer review, h-indexes and byzantine grant application forms.<p>And whenever something has prestige, people flock to it and want to also bask in it. And it gets inevitably diluted. But the prestige will move on and there will be some other thing next. Something we will call something else than &quot;science&quot;.