Answers.Body: <p>The problem with the current diffusion models, is that they are great density models, but they require to know a priori the size of the sample (ie the size of the image), which definitely is not the case for text</p>
  <p>However, you might get around this by first having a model, an encoder, that encodes text to a fixed size vector, and then a decoder that decodes that back to text, and use the diffusion model on the latest space of it</p>
  <p>However, most likely, it won't give great results, for the same reasons Seq2Seq is not the SoTA for text generation</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-06T21:18:33.120Z
Answers.LastActivityDate: 2024-06-06T21:18:33.120Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>Examples of language modelling using diffusion:</p>
  <ul>
  <li><a href="https://aclanthology.org/2023.emnlp-main.716.pdf" rel="nofollow noreferrer">CodeFusion: A Pre-trained Diffusion Model for Code Generation</a>. <a href="https://aclanthology.org/people/m/mukul-singh/" rel="nofollow noreferrer">Mukul Singh</a>, <a href="https://aclanthology.org/people/j/jose-cambronero/" rel="nofollow noreferrer">José Cambronero</a>, <a href="https://aclanthology.org/people/s/sumit-gulwani/" rel="nofollow noreferrer">Sumit Gulwani</a>, <a href="https://aclanthology.org/people/v/vu-le/" rel="nofollow noreferrer">Vu Le</a>, <a href="https://aclanthology.org/people/c/carina-negreanu/" rel="nofollow noreferrer">Carina Negreanu</a>, <a href="https://aclanthology.org/people/g/gust-verbruggen/" rel="nofollow noreferrer">Gust Verbruggen</a>. EMNLP 2023.</li>
  <li><a href="https://arxiv.org/abs/2112.06749" rel="nofollow noreferrer">Step-unrolled Denoising Autoencoders for Text Generation</a>. <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Savinov,+N" rel="nofollow noreferrer">Nikolay Savinov</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Chung,+J" rel="nofollow noreferrer">Junyoung Chung</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Binkowski,+M" rel="nofollow noreferrer">Mikolaj Binkowski</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Elsen,+E" rel="nofollow noreferrer">Erich Elsen</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=van+den+Oord,+A" rel="nofollow noreferrer">Aaron van den Oord</a>.  ICLR 2022.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-01T20:52:13.697Z
Answers.LastActivityDate: 2024-07-01T20:52:13.697Z
Answers.UserDisplayName: Franck Dernoncourt
Body: <p>Is there a popular diffusion model-based framework for language modelling? If not, is it because of the difficulty sampling for discrete distributions?</p>
  
Tags: <natural-language-processing><large-language-models><diffusion-models><language-model>
Title: Any popular diffusion model for language modeling?
UserDisplayName: Ayan Sengupta
ThreadText: # Any popular diffusion model for language modeling?
  
  (Asked by: username_0 on 2024-06-06)
  
  Is there a popular diffusion model-based framework for language modelling? If not, is it because of the difficulty sampling for discrete distributions?
  
  **Tags:** natural-language-processing, large-language-models, diffusion-models, language-model
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-06-06. Score: 1
  
  The problem with the current diffusion models, is that they are great density models, but they require to know a priori the size of the sample (ie the size of the image), which definitely is not the case for text
  
  However, you might get around this by first having a model, an encoder, that encodes text to a fixed size vector, and then a decoder that decodes that back to text, and use the diffusion model on the latest space of it
  
  However, most likely, it won't give great results, for the same reasons Seq2Seq is not the SoTA for text generation
  
  
  ---
  
  ### Answer by username_2 on 2024-07-01. Score: 0
  
  Examples of language modelling using diffusion:
  
  * [CodeFusion: A Pre-trained Diffusion Model for Code Generation](https://aclanthology.org/2023.emnlp-main.716.pdf). [Mukul Singh](https://aclanthology.org/people/m/mukul-singh/), [José Cambronero](https://aclanthology.org/people/j/jose-cambronero/), [Sumit Gulwani](https://aclanthology.org/people/s/sumit-gulwani/), [Vu Le](https://aclanthology.org/people/v/vu-le/), [Carina Negreanu](https://aclanthology.org/people/c/carina-negreanu/), [Gust Verbruggen](https://aclanthology.org/people/g/gust-verbruggen/). EMNLP 2023.
  * [Step-unrolled Denoising Autoencoders for Text Generation](https://arxiv.org/abs/2112.06749). [Nikolay Savinov](https://arxiv.org/search/cs?searchtype=author&query=Savinov,+N), [Junyoung Chung](https://arxiv.org/search/cs?searchtype=author&query=Chung,+J), [Mikolaj Binkowski](https://arxiv.org/search/cs?searchtype=author&query=Binkowski,+M), [Erich Elsen](https://arxiv.org/search/cs?searchtype=author&query=Elsen,+E), [Aaron van den Oord](https://arxiv.org/search/cs?searchtype=author&query=van+den+Oord,+A). ICLR 2022.
  
  
  ---
  
  