Comparing model evaluations of machine learning and statistics I'm a machine learning practitioner who spends most of my time doing applications via Python in machine vision with neural networks and remote sensing data, and evaluating model performance mostly using cross-validation based techniques etc.
Statistics has been (of course) part of my learning path and I am familiar with the basic concepts of statistics, but I am not actively doing the work of a "pure" statistician, that is, using e.g. R/SAS-software to fit generalized linear models, calculating the $t/p$-values, ANOVA and testing statistical significance of model covariates.
Even though I've been exposed to these fields for many years now, it still is not clear to me  how (traditional/pure) statistics and machine learning differ from each other. I know that the general concensus on the difference is that statistics cares more about explaining the data, whereas machine learning (ML) is interested in making predictions, even though the differences are sometimes vague and both fields use the same methods.
Now in ML, usually when we have fitted some model into the data, we care about how well the model has learned the underlying pattern from the data, and we measure this learning by using an independent test data set. We never want to have a perfect fit of the model into the training data, because this would lead into overfitting and poor generalization capability. So the relevant question in ML (to my understanding) is never about "how good the fit of the model is to the data?", but rather "how well the model is able to predict new situations, that is, generalize".
When I listen to my statistician colleagues, I notice them talking mostly about the $t/p$-values, the $R^2$-values, $F$-statistics, differences in the means of two groups in ANOVA etc.
To me, from a ML perspective, my statistician colleagues seem to be concerned exactly about the "goodness-of-fit" of their models/covariates into the data and there is no independent test data set, which they use to validate their models. Well, of course, they don't do this, because this is not the goal, but rather to explain the data with the explicit assumptions we have made on the data (normality etc. the usual ones).
Now, I most probably am wrong due to my lack of knowledge on the subject, but it somehow seems odd to me why"pure" statisticians are so interested in these "goodness-of-fit"-statistics, because don't the $p$-values, $R^2$ etc. essentially in the end measure exactly this, that is the model fit (given that our distribution assumptions are correct)? For example, we all know that neural networks are universal approximators, meaning we can fit them with 100% accuracy into any (continuous) function we arbitrarily choose by just adding enough neurons into the model. Now, wouldn't this universal approximating neural network tuned into our data like hell have a huge statistical significance in the $p$-values or $R^2$-metrics, if we look at the model fit from a "pure" statistician's perspective? As a summary, would a statistician (does he?) conclude that: "we have found something truly significant now" in this neural network scenario? A ML scientist would produce an independent test sample, feed it to the network, and conclude that the model is overfitting the training data as hell and no pattern has been found. In other words, ML scientist would conclude "nothing significant has been found".
Maybe I asked my question too vaguely, but here it is summarized: Is it true to some extent that statisticians are usually more concerned about the model's goodness-of-fit and the corresponding metrics of significance, and not that much about model's generalization capability,  and vice versa for the ML scientists?
Thank you in advance for any answers, which help me clear up my confusion :)

The question is very long, and covering it in full would require a very lengthy answer. So here I will try to provide my view with very brief bullet points.

*

*The question over-emphasises the difference between machine learning and statistics. For a good reflection on those I recommend reading the answer by Michael I. Jordan which was given during his Q&A reddit session: link


*You use "pure" statisticians in an unusual way. From my experience most "pure" statisticians don't care about concrete data at all, they are more interested in creating estimators and proving properties about these estimators. Applied statisticians (of which machine learning practitioners is one type) then use these estimators on concrete datasets in order to answer practical questions.


*"Pure" statistician come up with methods and then test how well those methods behave under various contexts. Applied statisticians then use well-behaved estimators that align with the properties of their data. Machine learning practitioners do the same, they use well-tested concepts like cross-validation (itself a product of statistics) to estimate the accuracy of their models.


*p-value is a measure of uncertainty, not accuracy. The concept itself is best understood as a form of enhanced induction, where you test a theory about the real world by only having observed a few facts. For example p-value can be applied to check how certain we are about our measure of accuracy on the test set.


*Accuracy cannot be a substitute for a p-value. Consider a scenario where we ask if there is a difference between two classes, and the overlap of those classes is 99%. Whatever you do with pure prediction you will only be able to get an accuracy of 51%. But with a big enough sample size you will reach an arbitrary small p-value, stating that those two classes are indeed different.


*Contrary to the statement that statisticians don't care about model generalisation - it's the opposite. Statisticians care about the generalisation of everything, not just accuracy. That's what things like confidence intervals and p-values try to achieve - to give a hint how well some estimate on a sample generalises to a broader population.


*In my personal opinion one of the bigger differences between the communities of statistics and machine learning (with exceptions of course) is the overall context of the effort. Statisticians place more emphasis on assumptions and selecting the best tool/model for the job at hand. You understand your data, check some assumptions, have a question, and device the best strategy of answering that single question and nothing else. While machine learning people place emphasis on the best overall possible method. For example, I would bet that a lot ML practitioners have a hope of achieving the perfect model that can mimic the human brain and can learn everything and solve multiple problems without being pre-trained, etc. and seems like that is what a big chunk of the community is working towards.
