TEncDM: Understanding the Properties of Diffusion Model in the Space of Language Model Encodings
<|reference_start|>TEncDM: Understanding the Properties of Diffusion Model in the Space of Language Model Encodings: This paper presents the Text Encoding Diffusion Model (TEncDM), a novel approach to diffusion modeling that operates in the space of pre-trained language model encodings. In contrast to traditionally used embeddings, encodings integrate contextual information. In our approach, we also employ a transformer-based decoder, specifically designed to incorporate context in the token prediction process. We conduct a comprehensive examination of the influence of the encoder, decoder, noise scheduler, and self-conditioning on zero-shot generation. Furthermore, we compare TEncDM with previous approaches on three conditional text generation tasks: QQP, XSum, and Wiki-Auto. The results show that TEncDM exhibits superior performance compared to existing non-autoregressive diffusion models.<|reference_end|>
@article{shabalin2024tencdm:,
    title={TEncDM: Understanding the Properties of Diffusion Model in the Space of
  Language Model Encodings},
    author={Alexander Shabalin, Viacheslav Meshchaninov, Egor Chimbulatov,
  Vladislav Lapikov, Roman Kim, Grigory Bartosh, Dmitry Molchanov, Sergey
  Markov, Dmitry Vetrov},
    journal={arXiv preprint arXiv:2402.19097},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.19097},
    primaryClass={cs.CL}
}