Answers.Body: <blockquote>
  <p>A foundation model is any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks</p>
  </blockquote>
  <p>from <a href="https://arxiv.org/abs/2108.07258" rel="noreferrer">&quot;On the Opportunities and Risks of Foundation Models&quot;</a> seems like a clear and reasonable definition.</p>
  <p><a href="https://dl.acm.org/doi/10.3115/981658.981684" rel="noreferrer">&quot;Unsupervised word sense disambiguation rivaling supervised methods&quot;</a> from 1995 is considered self-supervised. Most researchers would not call that part new to foundation models.</p>
  <p>The &quot;can be adapted to a wide range of downstream tasks&quot; also certainly pre-dates  modern &quot;foundation models&quot; e.g. <a href="https://dl.acm.org/doi/abs/10.1145/1273496.1273592" rel="noreferrer">&quot;Self-taught learning: transfer learning from unlabeled data&quot;</a> from 2007.</p>
  <p>So &quot;is it something fundamentally / truly new&quot;? Not really, but still very impressive.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-19T17:59:01.533Z
Answers.LastActivityDate: 2023-11-19T17:59:01.533Z
Answers.UserDisplayName: bogovicj
Body: <p>Currently, one can hear more and more about &quot;foundation models&quot; but details of this are not  always clear. Also, I even have the impression, that sometimes people don't talk about exactly the same thing. Sometimes all of this feels a bit nebulous in the spirit of &quot;<em>let's throw as much data as possible at as many GPUs as possible, and something useful and predictive will come out</em>&quot;. This clearly can be fun but in the worst case just produce a tremendous amount of CO2...</p>
  <p>Therefore I am wondering if there exists a clear definition of what a foundation model is (or is not) and if you know of good examples that demonstrate the usefulness and novelties? Do you believe that this is something truly new or just a scaled-up version of what we are doing anyway so far?</p>
  <p>Thanks in advance for sharing your insights!</p>
  
Tags: <deep-learning><transfer-learning>
Title: Are foundation models something fundamentally new? Is there a proper definition?
UserDisplayName: BanDoP
ThreadText: # Are foundation models something fundamentally new? Is there a proper definition?
  
  (Asked by: username_0 on 2023-11-14)
  
  Currently, one can hear more and more about "foundation models" but details of this are not always clear. Also, I even have the impression, that sometimes people don't talk about exactly the same thing. Sometimes all of this feels a bit nebulous in the spirit of "*let's throw as much data as possible at as many GPUs as possible, and something useful and predictive will come out*". This clearly can be fun but in the worst case just produce a tremendous amount of CO2...
  
  Therefore I am wondering if there exists a clear definition of what a foundation model is (or is not) and if you know of good examples that demonstrate the usefulness and novelties? Do you believe that this is something truly new or just a scaled-up version of what we are doing anyway so far?
  
  Thanks in advance for sharing your insights!
  
  **Tags:** deep-learning, transfer-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-19. Score: 5 | Accepted answer
  
  > A foundation model is any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks
  
  from ["On the Opportunities and Risks of Foundation Models"](https://arxiv.org/abs/2108.07258) seems like a clear and reasonable definition.
  
  ["Unsupervised word sense disambiguation rivaling supervised methods"](https://dl.acm.org/doi/10.3115/981658.981684) from 1995 is considered self-supervised. Most researchers would not call that part new to foundation models.
  
  The "can be adapted to a wide range of downstream tasks" also certainly pre-dates modern "foundation models" e.g. ["Self-taught learning: transfer learning from unlabeled data"](https://dl.acm.org/doi/abs/10.1145/1273496.1273592) from 2007.
  
  So "is it something fundamentally / truly new"? Not really, but still very impressive.
  
  
  ---
  
  