Foundation models : Is it a new paradigm for statistics and machine learning? A recent debate on so called Foundation models (CRFM) brings a real question of if we can build very large models on any specified domain, similar to current large language models, and replace our any statistical or machine learning modelling efforts to a tuning  library of existing foundation models exercise. Obviously, causality can not be addressed by these models but this approach would change how we practice statistics and data science in general. Is this a new paradigm for statistics and machine learning?
Edit
Making the question a bit less opinion based as raised by the community. More specifically, given the proposed foundation models, do we have similar supervised pretraining (or broadly speaking transfer learning) practices from the statistics literature that fitted models are used as a starting point in new tasks?
Edit
A recent article from Gradient's view Reflections on Foundation Models.
Edit
Psychology Today has published a short article AI's Paradigm Shift to Foundation Models.
Edit
A geophysics example Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark.
Edit
Along the similar lines, new architectures Google's PaLM and Open AI's DALL-E 2
Edit
A Generalist Agent from deepmind. Similar approach, "multi-domain learning".
Edit
ChatGPT: Optimizing Language Models for Dialogue

Disclosure: I didn't have time to carefully read the full paper yet.
From the abstract:

Despite the impending widespread deployment of foundation models,
we currently lack a clear understanding of how they work, when they fail, and what they are even
capable of due to their emergent properties.

Foundational models is a term with a good hype potential, but in reality there is nothing "foundational" in those models.

*

*The big language models can be fine-tunned or can serve as pre-trained layers of other models. It is not that they "replace" the other models in NLP. They don't solve "all" the problems.

*It is also not true that they are widespread. One thing is marketing and the companies that are trying to sell you those models, or sell you the costly infrastructure to run them. On another hand, a lot of problems is still sold with good old "shallow" machine learning, things like naive Bayes, rule-based algorithms, etc.

*There are results showing that "good old" models often outperform the big "SOTA" models, but nobody bothers to benchmark them anymore, but instead we go with the hype. One example are LSTMs.

*The big "foundational" models often fail miserably. When asked to generate text, they can produce complete nonsense (again, don't trust the cherry-picked marketing materials). One of such recent examples was the code generating GitHub Copilot by OpenAI, that was hugely overhyped, but in the end we learned that in 40% of cases it produces buggy code.

*Finally, the big NLP models exist mostly for English language. Good luck with finding equal quality models for other languages.

For me, the whole idea of "foundational models" is just "artificial general intelligence" idea in disguise. A lot of people still dream of building AGI, this didn't succeed, so now they're saying "Ok, let's make it almost-general". The problem is that we aren't there yet and we don't even know if we would ever reach it. At the same time, most of the real life problems are solved with relatively simple models, because they just work, are faster, cheaper, and usually easier to maintain.
