We're Incentivizing Bad Science
I don&#x27;t want to sound too triumphant, but over here in fundamental physics we have decades-old safeguards in place against all these problems.<p>The big experiments don&#x27;t have publication bias: they proudly say exactly what they did, even if 90% of the time there are only negative results, because exclusions are important too. Experiments are inherently replicated, with multiple independent simultaneous experiments (LHC) or multiple independent analyses (EHT), data blinding throughout, and even occasionally a further layer of blinding using decoy signals (LIGO). The statistical standards for discovery are, in terms of p-values, about 10,000 times more stringent, and even still people are moving away from p-values entirely.<p>The resulting publications are put out for free, publicly, on the ArXiv. Later they are submitted to a relatively small family of low-cost journals, which everybody knows the reputations of.<p>Hopefully some of these lessons can be adapted to other fields.