Comparing model evaluations of machine learning and statistics I'm a machine learning practitioner who spends most of my time doing applications via Python in machine vision with neural networks and remote sensing data, and evaluating model performance mostly using cross-validation based techniques etc.
Statistics has been (of course) part of my learning path and I am familiar with the basic concepts of statistics, but I am not actively doing the work of a "pure" statistician, that is, using e.g. R/SAS-software to fit generalized linear models, calculating the $t/p$-values, ANOVA and testing statistical significance of model covariates.
Even though I've been exposed to these fields for many years now, it still is not clear to me  how (traditional/pure) statistics and machine learning differ from each other. I know that the general concensus on the difference is that statistics cares more about explaining the data, whereas machine learning (ML) is interested in making predictions, even though the differences are sometimes vague and both fields use the same methods.
Now in ML, usually when we have fitted some model into the data, we care about how well the model has learned the underlying pattern from the data, and we measure this learning by using an independent test data set. We never want to have a perfect fit of the model into the training data, because this would lead into overfitting and poor generalization capability. So the relevant question in ML (to my understanding) is never about "how good the fit of the model is to the data?", but rather "how well the model is able to predict new situations, that is, generalize".
When I listen to my statistician colleagues, I notice them talking mostly about the $t/p$-values, the $R^2$-values, $F$-statistics, differences in the means of two groups in ANOVA etc.
To me, from a ML perspective, my statistician colleagues seem to be concerned exactly about the "goodness-of-fit" of their models/covariates into the data and there is no independent test data set, which they use to validate their models. Well, of course, they don't do this, because this is not the goal, but rather to explain the data with the explicit assumptions we have made on the data (normality etc. the usual ones).
Now, I most probably am wrong due to my lack of knowledge on the subject, but it somehow seems odd to me why"pure" statisticians are so interested in these "goodness-of-fit"-statistics, because don't the $p$-values, $R^2$ etc. essentially in the end measure exactly this, that is the model fit (given that our distribution assumptions are correct)? For example, we all know that neural networks are universal approximators, meaning we can fit them with 100% accuracy into any (continuous) function we arbitrarily choose by just adding enough neurons into the model. Now, wouldn't this universal approximating neural network tuned into our data like hell have a huge statistical significance in the $p$-values or $R^2$-metrics, if we look at the model fit from a "pure" statistician's perspective? As a summary, would a statistician (does he?) conclude that: "we have found something truly significant now" in this neural network scenario? A ML scientist would produce an independent test sample, feed it to the network, and conclude that the model is overfitting the training data as hell and no pattern has been found. In other words, ML scientist would conclude "nothing significant has been found".
Maybe I asked my question too vaguely, but here it is summarized: Is it true to some extent that statisticians are usually more concerned about the model's goodness-of-fit and the corresponding metrics of significance, and not that much about model's generalization capability,  and vice versa for the ML scientists?
Thank you in advance for any answers, which help me clear up my confusion :)

The answer is no, too many claims in your post, not only to the main question.

*

*The difference between ML and stats is arbitrary, superficial, and not important. There is plenty of statisticians that are doing predictive models where the main goal is, of course prediction. Common machine learning methods have been developed by statisticians and published in statistical journals. The most popular ML book is the elements of statistical learning, after all. Also, there is a huge subfield of ML that is concerned about the interpretation of fitted models and the effects of individual variables. I can tell you that (at least contemporary) papers discussing differences between ML and stats are cynical citation grabs and not really worth the time.


*Both ML and stats are concerned about learning the underlying pattern from the data. In stats, this might be called estimating effects, but it is the same thing.


*"[in ML] We never want to have a perfect fit of the model into the training data." Sometimes you do (search double descent)


*Statisticians do care about the generalization. I would even say more than ML people. All these statistical concepts, like controlling for type I error, confidence intervals, credible intervals etc., are there to tell you how well your findings generalize to the population. Now, if a machine learner tests 2 models in a test set and one is better than the other, this does not tell you much about how this finding will generalize to the population unless you calculate some standard errors, p-values or something on that difference. This is usually not the problem in computer vision with hundreds of thousands of examples, where pretty much any difference will be significant, but it is important in smaller problems.


*If a model has large goodness of fit because it is overfitted, p/t/F values will not be good, and statisticians will not find significant results in that scenario. In this case, variables will be colinear, standard errors will be huge, and nothing will be significant.


*in stats, we are often not using test sets because these statistical procedures had been developed to be valid without the need of the test set. If I use F-test to compare which model is better, the results will be generalization to the population because that's what the test is for. It will not automatically select the model with a higher R2.
In summary, statisticians are concerned about the generalization, and the common statistical measures and concepts are there for this purpose. On the other hand, ML people are often not interested in generalization because simple CV or test set performance is often not treated as an estimate of the performance in the population,
