This is a well-written article, and the concepts are explained clearly, thanks for sharing. I&#x27;d just like to add a caveat.<p>When the author says &quot;if a researcher runs 400 experiments on the same train-test splits&quot; - then depending on what he means by &#x27;test&#x27; set, that researcher is <i>wrong</i>. In pretty much all machine learning literature I&#x27;ve come across, it&#x27;s drilled into you that you <i>never look at your held-out test set</i> until the very end. Hyperparameter optimisation and&#x2F;or model selection happens on the training set and only when you&#x27;ve tuned your hyperparameters and selected your best model do you run the model on your test set to see how it&#x27;s done.<p>Once you&#x27;ve run the model on the test set once, you can&#x27;t go back to tweak your model because you&#x27;re introducing bias and you no longer have any data left that your model&#x27;s never seen before.<p>To avoid overfitting, you can use cross-validation to effectively re-use your training set and create multiple training&#x2F;validation splits. (As an aside, I find it frustrating how liberally different sources switch between &#x27;validation set&#x27; and &#x27;test set&#x27;, it&#x27;s really confusing).