Answers.Body: <p>I don't have a definitive answer but I'd state my intuitions anyways:</p>
  <p>Diffusion models are highly related to the idea of <strong>stacked denoising autoencoders</strong> <a href="https://www.researchgate.net/publication/272086159_Static_hand_gesture_recognition_using_stacked_Denoising_Sparse_Autoencoders" rel="noreferrer">[Kumar et al. (2014)]</a>. Additionally, U-Net-like architectures are a very common architecture for autoencoders on images.
  Here, I would agree with your intuition that the bottleneck and the skip-connections help with denoising because they provide representations of different granularity.</p>
  <p>Another thought is that U-Nets are empirically well-suited for image segmentation problems. Despite the fact that pixels are classified individually, you want the segmentation output to have consistent areas on top of objects in the image. So you kind of turn a very noisy segmentation mask (original image input) into a mask with much less noise (segmented output).</p>
  <p>I think the latter is debatable, I'd be happy to hear your thoughts.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-25T06:05:44.417Z
Answers.Text: Hi, thanks for reply. The "bottleneck" you mean is 1x1 conv or the process of lowering resolution? If it is the former, I saw an [article](https://benanne.github.io/2022/01/31/diffusion.html) that says diffusion model cares about output more than latent representations which I think is true. Maybe this is also one of the reasons why they remove bottleneck layer from architecture?  As for image segmentation, because I am not familiar with it, I am sorry that I cannot see the connection between them XD.
Answers.UserDisplayName: Penguin.jpg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-26T11:04:03.470Z
Answers.Text: With bottleneck I mean the decrease in layer size. This forces the encoding to discard some information which should improve denoising properties. The noise cannot be encoded lower dimensions.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-26T11:06:16.783Z
Answers.Text: Regarding the image segmentation part: Just look at [this image](https://nanonets.com/blog/content/images/2020/08/koeln00.png) and you know what image segmentation means. What you see there is the model output (colors) superimposed on the model input (a regular image). The goal of the model is to classify every pixel based on which object it belongs to (colors denote object classes). Thus, you could think of this task as the transformation of a very noisy version of this segmentation map (which is the regular image) to a very clean segmentation map. I hope this made it somewhat clearer?
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-26T11:19:39.053Z
Answers.Text: Yes, it is clearer, thanks for explanation! If that is the case, I think I prefer the former idea. Maybe it is really just a simple reason like this that supports them to use it :o.
Answers.UserDisplayName: Penguin.jpg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-14T05:45:15.923Z
Answers.Text: I think stable diffusion uses UNet because diffusion probabilistic models used UNet, DPM used UNet because PixelCNN++ used UNet, PixelCNN++ used it because it was 2017, that's what they had and it was good enough. Deep learning research has always been about cargo cult empirical research with weak (if any) justification. That said, the other choice is stuff like MaskRCNN and RPNs don't make any sense on latent representations
Answers.UserDisplayName: stuart
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-17T16:10:16.843Z
Answers.Text: I totally get your point and agree to some extend. I still think it is very fruitful to try to interpret a little bit more into why certain architectures are used in a given framework. This has led to great advances, e.g. the whole area of geometric deep learning is kind of an artifact of identifying the benefits of specific models and casting them into more general versions.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-24T17:44:16.193Z
Answers.LastActivityDate: 2022-09-24T17:44:16.193Z
Answers.UserDisplayName: Chillston
Answers.Body: <p><a href="https://arxiv.org/pdf/2212.09748.pdf" rel="nofollow noreferrer">They</a> <a href="https://arxiv.org/pdf/2209.12152.pdf" rel="nofollow noreferrer">do</a> <a href="https://arxiv.org/pdf/2208.07791.pdf" rel="nofollow noreferrer">not</a>.</p>
  <p>Even &quot;the&quot; <a href="https://arxiv.org/pdf/1503.03585.pdf" rel="nofollow noreferrer">seminal paper</a> does not (it was put on axiv before &quot;U-Net&quot; was a thing :).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-22T15:06:06.867Z
Answers.LastActivityDate: 2023-12-22T15:06:06.867Z
Answers.UserDisplayName: Warpig
Answers.Body: <p>Generally, U-Nets are the default choice when dealing with functions where inputs and outputs have the same size/shape (such as in image segmentation) due to the lack of a real information bottleneck. Thus, those are a natural good choice when dealing with diffusion models, where the network must predict the residual noise.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-14T09:11:58.697Z
Answers.LastActivityDate: 2024-03-14T09:11:58.697Z
Answers.UserDisplayName: akanai
Body: <p>I want to know why diffusion models always use U-Net.</p>
  <p>In my opinion, they use U-Net because you can see features of different resolutions and skip connection is good to add detail of images. But I am not sure if it is the main reason why they use U-Net.</p>
  <p>Are there other reasons they choose U-Net rather than other architectures?</p>
  
Tags: <machine-learning><computer-vision><generative-model><u-net>
Title: Why diffusion model always use U-Net?
UserDisplayName: Penguin.jpg
ThreadText: # Why diffusion model always use U-Net?
  
  (Asked by: username_0 on 2022-09-24)
  
  I want to know why diffusion models always use U-Net.
  
  In my opinion, they use U-Net because you can see features of different resolutions and skip connection is good to add detail of images. But I am not sure if it is the main reason why they use U-Net.
  
  Are there other reasons they choose U-Net rather than other architectures?
  
  **Tags:** machine-learning, computer-vision, generative-model, u-net
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2022-09-24. Score: 6
  
  I don't have a definitive answer but I'd state my intuitions anyways:
  
  Diffusion models are highly related to the idea of **stacked denoising autoencoders** [[Kumar et al. (2014)]](https://www.researchgate.net/publication/272086159_Static_hand_gesture_recognition_using_stacked_Denoising_Sparse_Autoencoders). Additionally, U-Net-like architectures are a very common architecture for autoencoders on images.
  Here, I would agree with your intuition that the bottleneck and the skip-connections help with denoising because they provide representations of different granularity.
  
  Another thought is that U-Nets are empirically well-suited for image segmentation problems. Despite the fact that pixels are classified individually, you want the segmentation output to have consistent areas on top of objects in the image. So you kind of turn a very noisy segmentation mask (original image input) into a mask with much less noise (segmented output).
  
  I think the latter is debatable, I'd be happy to hear your thoughts.
  
  > **Comments:**
  > **username_0**: Hi, thanks for reply. The "bottleneck" you mean is 1x1 conv or the process of lowering resolution? If it is the former, I saw an [article](https://benanne.github.io/2022/01/31/diffusion.html) that says diffusion model cares about output more than latent representations which I think is true. Maybe this is also one of the reasons why they remove bottleneck layer from architecture?  As for image segmentation, because I am not familiar with it, I am sorry that I cannot see the connection between them XD.  
  > **username_1**: With bottleneck I mean the decrease in layer size. This forces the encoding to discard some information which should improve denoising properties. The noise cannot be encoded lower dimensions.  
  > **username_1**: Regarding the image segmentation part: Just look at [this image](https://nanonets.com/blog/content/images/2020/08/koeln00.png) and you know what image segmentation means. What you see there is the model output (colors) superimposed on the model input (a regular image). The goal of the model is to classify every pixel based on which object it belongs to (colors denote object classes). Thus, you could think of this task as the transformation of a very noisy version of this segmentation map (which is the regular image) to a very clean segmentation map. I hope this made it somewhat clearer?  
  > **username_0**: Yes, it is clearer, thanks for explanation! If that is the case, I think I prefer the former idea. Maybe it is really just a simple reason like this that supports them to use it :o.  
  > **username_2**: I think stable diffusion uses UNet because diffusion probabilistic models used UNet, DPM used UNet because PixelCNN++ used UNet, PixelCNN++ used it because it was 2017, that's what they had and it was good enough. Deep learning research has always been about cargo cult empirical research with weak (if any) justification. That said, the other choice is stuff like MaskRCNN and RPNs don't make any sense on latent representations  
  > **username_1**: I totally get your point and agree to some extend. I still think it is very fruitful to try to interpret a little bit more into why certain architectures are used in a given framework. This has led to great advances, e.g. the whole area of geometric deep learning is kind of an artifact of identifying the benefits of specific models and casting them into more general versions.  
  
  ---
  
  ### Answer by username_3 on 2023-12-22. Score: 1
  
  [They](https://arxiv.org/pdf/2212.09748.pdf) [do](https://arxiv.org/pdf/2209.12152.pdf) [not](https://arxiv.org/pdf/2208.07791.pdf).
  
  Even "the" [seminal paper](https://arxiv.org/pdf/1503.03585.pdf) does not (it was put on axiv before "U-Net" was a thing :).
  
  
  ---
  
  ### Answer by username_4 on 2024-03-14. Score: 0
  
  Generally, U-Nets are the default choice when dealing with functions where inputs and outputs have the same size/shape (such as in image segmentation) due to the lack of a real information bottleneck. Thus, those are a natural good choice when dealing with diffusion models, where the network must predict the residual noise.
  
  
  ---
  
  