Machine Learning Reproducibility Checklist [pdf]
This checklist has some flaws. Most interesting results in ML have no proof.For example, can you give a proof of superconvergence? What’s the exact learning rate that causes it, and why? Did you know that you can often get away with a high learning rate for a time, and then divergence happens? What’s the proof of that?Give a proof that under all circumstances and wind conditions, lowering your airplane’s flaps by 5 degrees will help you land safely.Also, what about datasets that you’re not allowed to release? I personally despise such datasets, but I found myself in the ironic position of having a 10GB dataset dropped in my lap that was a perfect fit for my current project. Unfortunately it wasn’t until after training was mostly complete that we realized we hadn’t asked whether the author was comfortable releasing it, and indeed the answer was no. So what to do? Just don’t talk about it?I guess the list is good as a set of ideals to aim for. I just wish some consideration was given that you often can’t meet all of those goals.Most of OpenAI's work would be excluded by this checklist. I don't think anyone would argue that OpenAI doesn't do important work, and that their results are in some sense reproducible.