A Worrying Analysis of Recent Neural Recommendation Approaches
Summarizing, they tried replicating the results of 18 recent papers, and determined that because of lack of available code or data, they were only able to attempt replications of 7 of them.  Of these, they found that the "improvements" in 6 of them could also be achieved with a simpler non-machine learning approach (such as nearest neighbor). For the 7th (Mult-VAE), they were able to almost match the improvement  with better tuning of one of the baselines used in that paper.Does this mean that the published results are garbage?  No!  But what does it mean?First, don't miss the main claim:  less than half the published papers have sufficient code and data available for replication.   While one might argue exactly what the standards for availability should be, if journals and reviewers were to demand it, this number can definitely be improved.   Greater availability of data and code makes for better science.Second, my guess is that it means that paper authors are using excess "puffery" in an attempt to improve the chances that their papers are accepted for publication.  They've decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement.   Unfortunately, they are probably right.The useful change (in my opinion) would be to change the standards by which papers are judged to be publishable.  Reviewers should push back against weak baselines, but be more accepting of results that make modest claims.  You can still have a useful paper even if the degree of improvement is small --- or even nonexistent.  Maybe it will inspire someone else, maybe it's useful in an ensemble, or maybe it applies to a case where the baseline wouldn't.  It's the puffery that's the problem, not the publication.  Creating a system that rewards authors who more honestly evaluate their work would be an overall win.