Answers.Body: <p>At this point in time, there does not appear to be a really widely-agreed-upon definition of &quot;Foundation models&quot;. If you want one, the best place to go would be <a href="https://arxiv.org/abs/2108.07258" rel="noreferrer">this paper from Stanford that coined the term</a>. Generally at least some of the following ideas apply (according to some people they must all apply, according to others only some need to apply):</p>
  <ul>
  <li>Trained using unsupervised or self-supervised learning.</li>
  <li>Large (deep neural network) model.</li>
  <li>Not intended to be used directly for any particular end-goal.</li>
  <li>Intended to serve as a basis (&quot;foundation&quot;). You can think of this as &quot;warm-starting&quot;, starting from a well-trained initial model, that you can then fine-tune with further training (for example, supervised learning) for any specific task you personally have in mind.</li>
  <li>Trained on multimodal data (not just text, not just images, not just audio, etc., but a mix of things). <em>In my personal opinion this point really is not strictly necessary, and I think most people would agree, but I've seen some people saying this.</em></li>
  </ul>
  <p>Large Language Models would typically be trained specifically on language-related data (text). So, I suppose an LLM could sometimes serve as a Foundation model, but it's not necessarily the same thing.</p>
  <p><strong>NOTE</strong>: I would like to remark that, from many experts outside of Stanford, there is <a href="https://twitter.com/tdietterich/status/1558256704696905728" rel="noreferrer">quite a bit of pushback against the particular term &quot;Foundation model&quot;</a>. There seems to be a bit of PR involved, where it seems like they may like to coin a new term for something that really doesn't need a new term. Just saying it's, for example, a <a href="https://twitter.com/raphaelmilliere/status/1558460796321021953" rel="noreferrer">large pre-trained model</a>, may be more clear.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-15T20:01:42.453Z
Answers.LastActivityDate: 2022-08-15T20:01:42.453Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I read a lot about foundation model and large language model.</p>
  <p>However, I dont find a clear definition what exactly is a foundation model. Is large language model and foundation model the same thing?</p>
  
Tags: <deep-learning><terminology><large-language-models>
Title: Is large language model and foundation model the same thing?
UserDisplayName: Exploring
ThreadText: # Is large language model and foundation model the same thing?
  
  (Asked by: username_0 on 2022-08-15)
  
  I read a lot about foundation model and large language model.
  
  However, I dont find a clear definition what exactly is a foundation model. Is large language model and foundation model the same thing?
  
  **Tags:** deep-learning, terminology, large-language-models
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-08-15. Score: 9
  
  At this point in time, there does not appear to be a really widely-agreed-upon definition of "Foundation models". If you want one, the best place to go would be [this paper from Stanford that coined the term](https://arxiv.org/abs/2108.07258). Generally at least some of the following ideas apply (according to some people they must all apply, according to others only some need to apply):
  
  * Trained using unsupervised or self-supervised learning.
  * Large (deep neural network) model.
  * Not intended to be used directly for any particular end-goal.
  * Intended to serve as a basis ("foundation"). You can think of this as "warm-starting", starting from a well-trained initial model, that you can then fine-tune with further training (for example, supervised learning) for any specific task you personally have in mind.
  * Trained on multimodal data (not just text, not just images, not just audio, etc., but a mix of things). *In my personal opinion this point really is not strictly necessary, and I think most people would agree, but I've seen some people saying this.*
  
  Large Language Models would typically be trained specifically on language-related data (text). So, I suppose an LLM could sometimes serve as a Foundation model, but it's not necessarily the same thing.
  
  **NOTE**: I would like to remark that, from many experts outside of Stanford, there is [quite a bit of pushback against the particular term "Foundation model"](https://twitter.com/tdietterich/status/1558256704696905728). There seems to be a bit of PR involved, where it seems like they may like to coin a new term for something that really doesn't need a new term. Just saying it's, for example, a [large pre-trained model](https://twitter.com/raphaelmilliere/status/1558460796321021953), may be more clear.
  
  
  ---
  
  