Foundation models : Is it a new paradigm for statistics and machine learning? A recent debate on so called Foundation models (CRFM) brings a real question of if we can build very large models on any specified domain, similar to current large language models, and replace our any statistical or machine learning modelling efforts to a tuning  library of existing foundation models exercise. Obviously, causality can not be addressed by these models but this approach would change how we practice statistics and data science in general. Is this a new paradigm for statistics and machine learning?
Edit
Making the question a bit less opinion based as raised by the community. More specifically, given the proposed foundation models, do we have similar supervised pretraining (or broadly speaking transfer learning) practices from the statistics literature that fitted models are used as a starting point in new tasks?
Edit
A recent article from Gradient's view Reflections on Foundation Models.
Edit
Psychology Today has published a short article AI's Paradigm Shift to Foundation Models.
Edit
A geophysics example Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark.
Edit
Along the similar lines, new architectures Google's PaLM and Open AI's DALL-E 2
Edit
A Generalist Agent from deepmind. Similar approach, "multi-domain learning".
Edit
ChatGPT: Optimizing Language Models for Dialogue

The Bitter Lesson  is that in the long term, progress is dependent on leveraging more and more computational power. This is not to say that algorithmic and modeling progress isn't important, but they aren't the limiting factor -- neural networks have been since the 1950s (or earlier), and it's only now that increasing computation resources have let us exploit them fully.
The scaling hypothesis is the proposal that current models are only being held back by computation, and if we had several orders of magnitude more, we'd see dramatic improvements in modeling performance. This was explored and borne out by recent explorations into increasingly large language models.

(figure from here)
These recent large scale language models also demonstrate impressive few-shot or zero-shot capabilities, which validates the scaling hypothesis, and it sounds like the linked article concludes these "Foundation models" will come to replace more bespoke, individually trained models (although of course, no one is arguing that big models are going to replace the t-test).
Personally, I think there is a mountain of evidence for the bitter lesson, and for the scaling hypothesis, and these large language models are definitely very impressive. I don't have any opinion on whether this constitutes a new "paradigm" though (ideas like "the bitter lesson" have been floating around for many years, although the exploitation of supervised pretraining is relatively new), or whether these models will replace all others in the near future.
