What is the benefit of k-fold cross validation? I understand that using 100% of the dataset and doing k-fold cross validation instead of train_test_split would eliminate that randomness the latter method have in splitting and thus potentially avoid overfitting.
But I have seen that this is not the best practice when it comes to k-fold CV. What I have seen is that we should split the dataset (say 80% training, 20% testing), and then perform k-fold CV on the 80% training (Image below).
What I am confused about is that with this method, we are again potenially falling into a random split by using train_test_split before CV. Why is this method then considered best practice generally ? What am I missing ?


Generally we would prefer to have more data, not less. So we need to make efficient use of our data. That can leave you with smallish test datasets. If you have one 20% test dataset you do not know, how precisely that describes the predictive value. If you have 5 different 20% test data Sets you can get a rough idea of whether these lead to 5 similar or 5 diverging result.
There is always leave-one-out on the other extreme but that gets computationally challenging very fast.
