Answers.Body: <p>Diffusion models are very data hungry. Without data augmentation that won't be enough to train it nor would I expect realistic images from a dataset that small for something so complicated. Diffusion models also are very bad at producing noisy images, so you are going to have to train it on a set of images that are less noisy.</p>
  <p>My suggestion?</p>
  <ul>
  <li>More Data and make it clean. The more complicated, the more data you need. The original paper used millions to get realistic images. I would say at a minimum 20k is barely enough to see reasonable results without overfitting.</li>
  <li>Make sure you self attention is using convolutions instead of MLP's. This is what they call &quot;attention&quot; if you look at the implementation on the official DDIM repo.</li>
  <li>Adjust hyper parameters. Diffusion models can sometimes be very sensitive to hyper parameters. It's not easy like a VAE but not as hard as GAN. Try some more fiddling.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-29T05:26:24.443Z
Answers.Text: That's probably the issue. 10K images is not much and they are quite complex compared to the mnist number set.
Answers.UserDisplayName: zlobi.wan.kenobi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-23T02:48:28.787Z
Answers.LastActivityDate: 2023-12-23T02:48:28.787Z
Answers.UserDisplayName: Kode
Answers.Body: <p>To assess whether your model is learning, you can monitor the loss function during training. If the loss is decreasing over time, it indicates that the model is making progress. Additionally, you can visually inspect the generated images at different training stages to see if they progressively resemble motorbikes more accurately.</p>
  <p>You can employ techniques such as early stopping, where training is stopped if there is no improvement in validation loss over a certain number of epochs. Additionally, you can use validation metrics to assess the performance of the model on a separate validation set and stop training when the performance reaches a satisfactory level.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-11T14:02:14.387Z
Answers.Text: This does not answer the question. Also looks a lot like this is raw output from ChatGPT or similar. Unfortunately for you, it is not possible to "rep farm" using ChatGPT, because it is exceptionally bad at answering the kinds of questions that get asked on Stack Exchange. You can still use ChatGPT to help with your writing style, but you are going to have to monitor and understand its output, then correct its mistakes. Otherwise you will just be generating low-quality noise on the site
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-11T13:26:47.290Z
Answers.LastActivityDate: 2023-12-11T13:26:47.290Z
Answers.UserDisplayName: han
Body: <p>I am trying to train a diffusion model (from scratch in pytorch). UNet used is not anything too fancy, takes in images and time step as input for about 512 time steps. I am using learnable embeddings for 512 time steps. This works just fine for mnist but when moving to another dataset (my dataset is of motorbikes) it produces nothing meaningful even after 20k steps (batchsize 32). This is what i get after 20k steps on it. <a href="https://i.sstatic.net/LSKRe.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LSKRe.png" alt="motorbike results" /></a></p>
  <p>Some experiments i tried somewhat contradict each other and i cant think of why this is happening.</p>
  <p>Experiment:</p>
  <ol>
  <li>overfit a small batch of my dataset. Result: model produces something intangible like the above picture</li>
  <li>Have a dataset of about 10k images. Produces something like a bike if you really look for it. Trained this for ~28k steps (batchsize 32). <a href="https://i.sstatic.net/1QHmV.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1QHmV.jpg" alt="bikes produced by model" /></a></li>
  <li>model works fine for mnist and cifar (even after running it for a short while). <a href="https://i.sstatic.net/Lcqs7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Lcqs7.png" alt="mnist results" /></a></li>
  <li>Noticed there was noisy data which has nothing to do with bikes, so i manually got rid of them hoping my model would perform better. (reduced my dataset of 10k images to ~6.5k clean, quality images). Result: No changes, produces intangible images (just like first one) after training for ~55k steps (batchsize 32). This surprises me because i was expecting it to atleast be like the images produced from the noisy dataset.</li>
  </ol>
  <p>I have not changed model hyperparameters when performing these experiments:</p>
  <ol>
  <li>lr = 4e-4 with Adam</li>
  <li>batch_size = 32 (for bikes)</li>
  <li>Normalised images from [-1,1]</li>
  <li>img_size = (3, 128, 128) # c, h, w</li>
  <li>timesteps = 512</li>
  <li>betas scheduler from 10e-4 to 0.02 (just like in the original paper)</li>
  <li>using a cosine scheduler (this works a bit better than using a linear one)</li>
  </ol>
  <p>I tried using self attention but that performs worse (not sure if its &quot;worse&quot; because it produces noise even after 12.5k steps on mnist (batchsize of 64).</p>
  <p>I could not find a practical guide on training diffusion models, however from the information i gathered, the takeaway is that you have to train these forever and wont really work well if you run it for few hours on a mediocre gpu. I see that some of the sota diffusion models were trained on a few thousand days worth of gpu compute. Is my assumption correct ?</p>
  <p>I am still not clear on why overfitting on a batch does not work here or why the clean dataset does not produce anything tangible. How do i know if my model is ever learning ? when do i know i should stop ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-05T17:21:19.447Z
Comments.Text: Please, share code of at least your model structure or the Github link if you uploaded. Without seeing the program, it is hard to come out with suggestion. Also regarding your training dataset, do you make your own or download from the Internet? Please, add details of your dataset like total number of image, characteristic, average image size, and etc.
Comments.UserDisplayName: Cloud Cho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-29T01:43:38.143Z
Comments.Text: Did you find any solution to this? I have the exact same issue. With a low number of data points (2k), the model can overfit or produce some meaningful results after 4k steps. However, with a higher number of data points (180k), the model still produces very poor results after 60k steps, similar to what you described for bicycles. My findings up to this point are:
  1. Hyperparameters are very important; with a learning rate of 1e-06 (the default for text-to-image in the Diffuser sample), the model can't even learn from a small dataset of 2k. Setting the learning rate to 5e-5 works better.
  2. Lo
Comments.UserDisplayName: Hassan Hamidi
Tags: <deep-learning><generative-model><image-generation><embeddings><diffusion-models>
Title: Do diffusion models take a long time to train?
UserDisplayName: Aditya 
ThreadText: # Do diffusion models take a long time to train?
  
  (Asked by: username_0 on 2023-12-05)
  
  I am trying to train a diffusion model (from scratch in pytorch). UNet used is not anything too fancy, takes in images and time step as input for about 512 time steps. I am using learnable embeddings for 512 time steps. This works just fine for mnist but when moving to another dataset (my dataset is of motorbikes) it produces nothing meaningful even after 20k steps (batchsize 32). This is what i get after 20k steps on it. [![motorbike results](https://i.sstatic.net/LSKRe.png)](https://i.sstatic.net/LSKRe.png)
  
  Some experiments i tried somewhat contradict each other and i cant think of why this is happening.
  
  Experiment:
  
  1. overfit a small batch of my dataset. Result: model produces something intangible like the above picture
  2. Have a dataset of about 10k images. Produces something like a bike if you really look for it. Trained this for ~28k steps (batchsize 32). [![bikes produced by model](https://i.sstatic.net/1QHmV.jpg)](https://i.sstatic.net/1QHmV.jpg)
  3. model works fine for mnist and cifar (even after running it for a short while). [![mnist results](https://i.sstatic.net/Lcqs7.png)](https://i.sstatic.net/Lcqs7.png)
  4. Noticed there was noisy data which has nothing to do with bikes, so i manually got rid of them hoping my model would perform better. (reduced my dataset of 10k images to ~6.5k clean, quality images). Result: No changes, produces intangible images (just like first one) after training for ~55k steps (batchsize 32). This surprises me because i was expecting it to atleast be like the images produced from the noisy dataset.
  
  I have not changed model hyperparameters when performing these experiments:
  
  1. lr = 4e-4 with Adam
  2. batch\_size = 32 (for bikes)
  3. Normalised images from [-1,1]
  4. img\_size = (3, 128, 128) # c, h, w
  5. timesteps = 512
  6. betas scheduler from 10e-4 to 0.02 (just like in the original paper)
  7. using a cosine scheduler (this works a bit better than using a linear one)
  
  I tried using self attention but that performs worse (not sure if its "worse" because it produces noise even after 12.5k steps on mnist (batchsize of 64).
  
  I could not find a practical guide on training diffusion models, however from the information i gathered, the takeaway is that you have to train these forever and wont really work well if you run it for few hours on a mediocre gpu. I see that some of the sota diffusion models were trained on a few thousand days worth of gpu compute. Is my assumption correct ?
  
  I am still not clear on why overfitting on a batch does not work here or why the clean dataset does not produce anything tangible. How do i know if my model is ever learning ? when do i know i should stop ?
  
  **Tags:** deep-learning, generative-model, image-generation, embeddings, diffusion-models
  > **Comments:**
  > **username_1**: Please, share code of at least your model structure or the Github link if you uploaded. Without seeing the program, it is hard to come out with suggestion. Also regarding your training dataset, do you make your own or download from the Internet? Please, add details of your dataset like total number of image, characteristic, average image size, and etc.  
  > **username_2**: Did you find any solution to this? I have the exact same issue. With a low number of data points (2k), the model can overfit or produce some meaningful results after 4k steps. However, with a higher number of data points (180k), the model still produces very poor results after 60k steps, similar to what you described for bicycles. My findings up to this point are: 1. Hyperparameters are very important; with a learning rate of 1e-06 (the default for text-to-image in the Diffuser sample), the model can't even learn from a small dataset of 2k. Setting the learning rate to 5e-5 works better. 2. Lo  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2023-12-23. Score: 2
  
  Diffusion models are very data hungry. Without data augmentation that won't be enough to train it nor would I expect realistic images from a dataset that small for something so complicated. Diffusion models also are very bad at producing noisy images, so you are going to have to train it on a set of images that are less noisy.
  
  My suggestion?
  
  * More Data and make it clean. The more complicated, the more data you need. The original paper used millions to get realistic images. I would say at a minimum 20k is barely enough to see reasonable results without overfitting.
  * Make sure you self attention is using convolutions instead of MLP's. This is what they call "attention" if you look at the implementation on the official DDIM repo.
  * Adjust hyper parameters. Diffusion models can sometimes be very sensitive to hyper parameters. It's not easy like a VAE but not as hard as GAN. Try some more fiddling.
  
  > **Comments:**
  > **username_4**: That's probably the issue. 10K images is not much and they are quite complex compared to the mnist number set.  
  
  ---
  
  ### Answer by username_5 on 2023-12-11. Score: -1
  
  To assess whether your model is learning, you can monitor the loss function during training. If the loss is decreasing over time, it indicates that the model is making progress. Additionally, you can visually inspect the generated images at different training stages to see if they progressively resemble motorbikes more accurately.
  
  You can employ techniques such as early stopping, where training is stopped if there is no improvement in validation loss over a certain number of epochs. Additionally, you can use validation metrics to assess the performance of the model on a separate validation set and stop training when the performance reaches a satisfactory level.
  
  > **Comments:**
  > **username_6**: This does not answer the question. Also looks a lot like this is raw output from ChatGPT or similar. Unfortunately for you, it is not possible to "rep farm" using ChatGPT, because it is exceptionally bad at answering the kinds of questions that get asked on Stack Exchange. You can still use ChatGPT to help with your writing style, but you are going to have to monitor and understand its output, then correct its mistakes. Otherwise you will just be generating low-quality noise on the site  
  
  ---
  
  