Answers.Body: <p>It's possible to mix and match all sorts of encoders and decoders. If the output of the encoder can be mapped to the input of the decoder, and a loss function can be backpropagated through the model, then it is possible to combine them.</p>
  <p>Image segmentation, however, can be done simply with U-NET, as it can be trained as an image segmentation model. You can use an encoder to 'encode' your image, to make it easier to segment with the U-NET. Im assuming that the Image-Segmentation library you linked is actually just doing that, using an encoder and then applying U-NET for image segmentation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-22T09:23:48.750Z
Answers.LastActivityDate: 2023-02-22T09:29:42.233Z
Answers.LastEditDate: 2023-02-22T09:29:42.233Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I had a conceptual questions regarding architectures. I am using this <a href="https://github.com/qubvel/segmentation_models.pytorch" rel="nofollow noreferrer">git hub repository</a> that allows one to quickly put together a segmentation pipeline. In reading the readme one thing that has me confused is separation of the encoder and decoders in the code base. I am using Unet which is listed as a decoder and I have resnet as my encoder.</p>
  <p>This confuses me because the Unet paper states that it has both an encoder and decoder. Is it possible to simply mix and match as long as the layers are compatible. Lastly if I am using a pre-trained encoder with a UNet decoder am I only training half the model or just giving it a head start? I ask because I am able to do much larger image sizes with this splitting as oppose just having a pure Unet.</p>
  <p>Thanks in advance!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-22T09:25:13.917Z
Comments.Text: welcome to this SE :)
Comments.UserDisplayName: Robin van Hoorn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-22T09:26:45.137Z
Comments.Text: Ive never heard of projects splitting up a U-NET, because it kind of ruins the idea of U-NET. Are you sure you are using only the decoder part? And not just the complete U-NET architecture?
Comments.UserDisplayName: Robin van Hoorn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-22T16:28:53.710Z
Comments.Text: @RobinvanHoorn per the git hub unless I am misunderstanding what they are doing they are only using the UNet architecture for the decoding. That package actually does not let you train a model from scratch far as I am aware. I am new so I could be misreading.
Comments.UserDisplayName: TheCodeNovice
Tags: <autoencoders><semantic-segmentation><encoder-decoder>
Title: How does mixing and matching encoders and decoders work in image segmentation?
UserDisplayName: TheCodeNovice
ThreadText: # How does mixing and matching encoders and decoders work in image segmentation?
  
  (Asked by: username_0 on 2023-02-21)
  
  I had a conceptual questions regarding architectures. I am using this [git hub repository](https://github.com/qubvel/segmentation_models.pytorch) that allows one to quickly put together a segmentation pipeline. In reading the readme one thing that has me confused is separation of the encoder and decoders in the code base. I am using Unet which is listed as a decoder and I have resnet as my encoder.
  
  This confuses me because the Unet paper states that it has both an encoder and decoder. Is it possible to simply mix and match as long as the layers are compatible. Lastly if I am using a pre-trained encoder with a UNet decoder am I only training half the model or just giving it a head start? I ask because I am able to do much larger image sizes with this splitting as oppose just having a pure Unet.
  
  Thanks in advance!
  
  **Tags:** autoencoders, semantic-segmentation, encoder-decoder
  
  > **Comments:**
  > **username_1**: welcome to this SE :)  
  > **username_1**: Ive never heard of projects splitting up a U-NET, because it kind of ruins the idea of U-NET. Are you sure you are using only the decoder part? And not just the complete U-NET architecture?  
  > **username_0**: @username_1 per the git hub unless I am misunderstanding what they are doing they are only using the UNet architecture for the decoding. That package actually does not let you train a model from scratch far as I am aware. I am new so I could be misreading.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-22. Score: 1 | Accepted answer
  
  It's possible to mix and match all sorts of encoders and decoders. If the output of the encoder can be mapped to the input of the decoder, and a loss function can be backpropagated through the model, then it is possible to combine them.
  
  Image segmentation, however, can be done simply with U-NET, as it can be trained as an image segmentation model. You can use an encoder to 'encode' your image, to make it easier to segment with the U-NET. Im assuming that the Image-Segmentation library you linked is actually just doing that, using an encoder and then applying U-NET for image segmentation.
  
  
  ---
  
  