Answers.Body: <p>The partial implementation you mentioned is likely using a strategy where the model predicts the noise and then subtracts it from the noisy input embedding to recover the clean denoised embedding, which is a valid approach. Both methods as predicting noise vs. predicting denoised embeddings directly (e.g., DDPM) are related and can be valid since they ultimately aim to recover the same denoised embeddings equivalently.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-28T07:14:32.787Z
Answers.LastActivityDate: 2024-11-28T07:14:32.787Z
Answers.UserDisplayName: cinch
Body: <p>I have recently been trying to understand the <a href="https://arxiv.org/pdf/2211.15089" rel="nofollow noreferrer">CDCD framework</a>. I am confused as to weather the authors propose to predict with the model the noise in the input embeddings, or denoised embeddings. From what I understand, the model would be predicting denoised embeddings.</p>
  <p>However, I also found <a href="https://github.com/elyxlz/cdcd-pytorch" rel="nofollow noreferrer">this partial implementation</a>, in which the scheduler for sampling is set to make predictions of type &quot;epsilon&quot;, which would be in fact noise in the input embeddings.</p>
  <p>This made me really confused. Am I missing something in the implementation, or just interpreting wrongly the paper?</p>
  
Tags: <neural-networks><generative-model><diffusion-models><categorical-data>
Title: What does the CDCD (Continuous Diffusion for Categorical Data) framework predict? Noise? Sample?
UserDisplayName: Rafa
ThreadText: # What does the CDCD (Continuous Diffusion for Categorical Data) framework predict? Noise? Sample?
  
  (Asked by: username_0 on 2024-11-27)
  
  I have recently been trying to understand the [CDCD framework](https://arxiv.org/pdf/2211.15089). I am confused as to weather the authors propose to predict with the model the noise in the input embeddings, or denoised embeddings. From what I understand, the model would be predicting denoised embeddings.
  
  However, I also found [this partial implementation](https://github.com/elyxlz/cdcd-pytorch), in which the scheduler for sampling is set to make predictions of type "epsilon", which would be in fact noise in the input embeddings.
  
  This made me really confused. Am I missing something in the implementation, or just interpreting wrongly the paper?
  
  **Tags:** neural-networks, generative-model, diffusion-models, categorical-data
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-28. Score: 0
  
  The partial implementation you mentioned is likely using a strategy where the model predicts the noise and then subtracts it from the noisy input embedding to recover the clean denoised embedding, which is a valid approach. Both methods as predicting noise vs. predicting denoised embeddings directly (e.g., DDPM) are related and can be valid since they ultimately aim to recover the same denoised embeddings equivalently.
  
  
  ---
  
  