Why do most ML papers totally ignore the variance of the validation error? In ML we're indoctrinated about the train/validation/test split. A simple split will provide a point estimate of the error for validation and one for test. If a k-fold approach is used we might get a couple more points and a rudimentary sense of mean and variance of the error distribution.
While the practical aspects of the approach seem clear (many modern models can take hours/days to train and we cannot afford to run a 50-fold cross validation), I still fail to see how many ML papers (or, for that matter, applied business applications) can get away with claiming decimal level improvements on a single error point estimate from a hold out.
Wouldn't it make sense to at least run some sort of bootstrap to assess the error distribution to be able to claim "our new model outperforms SOTA with 75% likelihood"? I mean, that doesn't sound as cool as "our model outperforms SOTA", but wouldn't the former be the right expectation for the scientific community?

The current trend in machine learning research is to train huge models. Let me quote one article

At OpenAI, an important machine-learning think tank, researchers recently designed and trained a much-lauded deep-learning language system called GPT-3 at the cost of more than $4 million. Even though they made a mistake when they implemented the system, they didn't fix it, explaining simply in a supplement to their scholarly publication that "due to the cost of training, it wasn't feasible to retrain the model."

If a model is to expensive to train to fix a bug, certainly nobody is going to bother with $k$-fold cross-validation.
Moreover, if you have a huge dataset and subsample it, the subsample would be still huge enough to drive the variance low. Of course assuming that re-training would be deterministic, what is not true as there are known examples of non-reproducible results in machine learning, or getting different results with different random seeds.
