Answers.Body: <p>It sounds like you're talking about KNN-LMs and Retrieval Augmented Generation.</p>
  <h3><a href="https://arxiv.org/pdf/1911.00172.pdf" rel="nofollow noreferrer">KNN-LMs</a></h3>
  <p>These are models that, during inference time, look up the embedding representation of their context in a datastore. They use the retrieved next-token-prediction in addition to their regular output to make the final prediction.</p>
  <p><a href="https://i.sstatic.net/EvO4x.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/EvO4x.jpg" alt="KNN-LM diagram from https://arxiv.org/pdf/1911.00172.pdf" /></a>
  The above figure is from the KNN-LM paper.</p>
  <h3>Retrieval Augmented Models</h3>
  <p>This is an extremely general class of models, which basically involve the retrieval of some information which gets added to the context before the model proceeds with regular autoregressive generation.</p>
  <p>This retrieved information can be web pages (e.g., Bing Chat), documents from an existing database (see: open-domain question answering), or information from a knowledge graph. <a href="https://arxiv.org/abs/2305.18846" rel="nofollow noreferrer">This paper</a> does knowledge graph retrieval, then adds an <em>encoded</em> subgraph into the context.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-30T21:35:53.510Z
Answers.LastActivityDate: 2023-09-30T21:35:53.510Z
Answers.UserDisplayName: Alexander Wan
Answers.Body: <p>External memory indexed from attention - <a href="https://arxiv.org/pdf/2112.04426.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2112.04426.pdf</a></p>
  <p>Regarding minimizing cost of autoregression, one could do encoder-decoder (eg T5) architecture, <a href="https://aclanthology.org/2023.sustainlp-1.6.pdf" rel="nofollow noreferrer">but with a twist</a>: Make the decoder <em>deliberately</em> much smaller network than the encoder which is doing most of the heavy lifting, but running only once. Such asymmetry works on assumption that &quot;speaking&quot; a concept from the latent space vector is a simpler task compared to &quot;understanding&quot; it done in the encoder.</p>
  <p>If this paper is correct, this allows you to run T5-FLAN at 6x inference speed at a cost of marginal (&lt;5%) loss. The paper utilizes pruning (ie downsizing pre-trained model), better scaling might be possible if trained like this from scratch.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-30T11:19:02.377Z
Answers.LastActivityDate: 2023-09-30T11:43:48.333Z
Answers.LastEditDate: 2023-09-30T11:43:48.333Z
Answers.UserDisplayName: kat
Body: <p>Large language models like GPT have been really successful lately. One downside is that they require a huge amount of resources to train, and still a lot of resources for inference, such that most models can't run inference on consumer hardware. Another problem is that they do not have an explicit world model, so that you can not be 100% sure if a response is factual or hallucinated.</p>
  <p>Along the old <a href="https://en.wikipedia.org/wiki/Neats_and_scruffies" rel="nofollow noreferrer">neat-scruffy axis</a>, I wonder if there are alternative architectures that are a bit more in the scruffy direction. For example, I could imagine that you use deep learning only for &quot;language understanding&quot;, but for &quot;world knowledge&quot; you would have a fact database like a triplestore. The language model of such a system would be much smaller, only needing to reproduce basic grammar, and the facts would be explicit instead of implicit.</p>
  <p>So are there any - potentially competitive - NLP models that are not of the autoregressive transformer type, scruffier, smaller, and yet operating in a similar space to GPT-3, LLaMA and co.?</p>
  
Tags: <gpt><large-language-models>
Title: Smaller, competitive LLMs that are not autoregressive transformers?
UserDisplayName: jdm
ThreadText: # Smaller, competitive LLMs that are not autoregressive transformers?
  
  (Asked by: username_0 on 2023-03-17)
  
  Large language models like GPT have been really successful lately. One downside is that they require a huge amount of resources to train, and still a lot of resources for inference, such that most models can't run inference on consumer hardware. Another problem is that they do not have an explicit world model, so that you can not be 100% sure if a response is factual or hallucinated.
  
  Along the old [neat-scruffy axis](https://en.wikipedia.org/wiki/Neats_and_scruffies), I wonder if there are alternative architectures that are a bit more in the scruffy direction. For example, I could imagine that you use deep learning only for "language understanding", but for "world knowledge" you would have a fact database like a triplestore. The language model of such a system would be much smaller, only needing to reproduce basic grammar, and the facts would be explicit instead of implicit.
  
  So are there any - potentially competitive - NLP models that are not of the autoregressive transformer type, scruffier, smaller, and yet operating in a similar space to GPT-3, LLaMA and co.?
  
  **Tags:** gpt, large-language-models
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-09-30. Score: 0
  
  It sounds like you're talking about KNN-LMs and Retrieval Augmented Generation.
  
  ### [KNN-LMs](https://arxiv.org/pdf/1911.00172.pdf)
  
  These are models that, during inference time, look up the embedding representation of their context in a datastore. They use the retrieved next-token-prediction in addition to their regular output to make the final prediction.
  
  [![KNN-LM diagram from https://arxiv.org/pdf/1911.00172.pdf](https://i.sstatic.net/EvO4x.jpg)](https://i.sstatic.net/EvO4x.jpg)
  The above figure is from the KNN-LM paper.
  
  ### Retrieval Augmented Models
  
  This is an extremely general class of models, which basically involve the retrieval of some information which gets added to the context before the model proceeds with regular autoregressive generation.
  
  This retrieved information can be web pages (e.g., Bing Chat), documents from an existing database (see: open-domain question answering), or information from a knowledge graph. [This paper](https://arxiv.org/abs/2305.18846) does knowledge graph retrieval, then adds an *encoded* subgraph into the context.
  
  
  ---
  
  ### Answer by username_2 on 2023-09-30. Score: 0
  
  External memory indexed from attention - <https://arxiv.org/pdf/2112.04426.pdf>
  
  Regarding minimizing cost of autoregression, one could do encoder-decoder (eg T5) architecture, [but with a twist](https://aclanthology.org/2023.sustainlp-1.6.pdf): Make the decoder *deliberately* much smaller network than the encoder which is doing most of the heavy lifting, but running only once. Such asymmetry works on assumption that "speaking" a concept from the latent space vector is a simpler task compared to "understanding" it done in the encoder.
  
  If this paper is correct, this allows you to run T5-FLAN at 6x inference speed at a cost of marginal (<5%) loss. The paper utilizes pruning (ie downsizing pre-trained model), better scaling might be possible if trained like this from scratch.
  
  
  ---
  
  