Answers.Body: <p>Without experimental evidence to back me up, I can not answer this with 100% confidence. However, I am fairly certain that this will cause issues depending on the model.</p>
  
  <p>U-net is essentially an auto-encoder, and due to the fact that it is all just one big neural network, it is likely it will learn the easiest pattern (as all NN do), and that is to find one single instance of an object and shade that region.</p>
  
  <p>Now why does this depend on the model? Well let's say you are using something slightly different, where region proposals are generated by a determinisitc algorithm we've predefined, then these regions are run through a CNN to segment them. In this case, as each region is without context of the entire image, the difference between 2 objects in an image and 1 is indistinguishable to the network (as regions may overlap), and as such, only using images with 1 object will not pose any problems (there is a name for early models like these, though it escapes me).</p>
  
  <p>So assuming I am correct, what should you do? The models that use a deterministic algorithm for region proposals are slow and old, so I wouldn't suggest that. Instead, I would think that you should first do some testing, to see if it actually does cause issues. Assuming it does, a good option could be to tamper with the training data and separate segments by a few pixels to sort of "force" multiple objects into existence.</p>
  
  <p>Regardless of such, I would still suggest using U-net. Fixing this issue (if it does arrise) should be relatively easy to do, so there's little to lose by using U-net and just trying the training.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-14T03:26:01.607Z
Answers.LastActivityDate: 2020-01-14T03:26:01.607Z
Answers.UserDisplayName: Recessive
Answers.Body: <p>I'm not aware of a model that could generalize from single segment to multi segment without nontrivial surgery, augmentation, and retraining, but I have had some success doing this in cases where the regions to segment are non-overlapping: After finding the first segment, edit the input image to distort or obscure the segmented region and run it through again, repeating until it stops finding new segments.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-09T16:47:32.917Z
Answers.LastActivityDate: 2021-03-09T16:47:32.917Z
Answers.UserDisplayName: Andy
Answers.Body: <p>In the problem of <code>Semantic Segmentation</code> one solves the problem of investigating, whether a particular pixel on the image belongs to a certain class. For example, on the photo taken from the street one may be interested , whether this pixel belongs to the road, traffic light, sign, people or background of no interest.</p>
  <p>Supeficially, in this setting it doesn't matter, whether there is a single instance of an object or multiple, and the thing, which is more important is the abundance of a particular class on an image.</p>
  <p>However, <code>CNN</code> adopt to a particular patterns, pertinent to the distribution, from which the data comes from. It can be the case, that a network has captured some property and tooks a pattern, which is relevant only for one instance case. Images with multiple instances of class of interest are like outliers in the data, and may be treated due to this incorrecty.</p>
  <p>I am not sure, whether it makes sense in yout particular problem, but maybe it is worth performing a <code>mosaic</code> augmentation <a href="https://towardsdatascience.com/data-augmentation-in-yolov4-c16bd22b2617" rel="nofollow noreferrer">https://towardsdatascience.com/data-augmentation-in-yolov4-c16bd22b2617</a>. So that you will have for some samples of the augmented data a multiinstance case.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-07T12:46:39.553Z
Answers.Text: I think you misunderstood the question. The question is not about the difference between semantic and instance segmentation. The question is whether the model would generalize and be able to segment images with a different number of objects than the number of objects that can be found in the training data. I think the OP is not interested in knowing whether the model would be able to differentiate the instances of the same type of object (i.e. they were not asking about instance segmentation). So, I don't think that this answer really addresses the question.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-07T13:51:26.587Z
Answers.Text: @nbro I mean that superficially due to the absence of notion of single or multiple instances of some class the task is the same. However, if in the training data there are instances with only single intstance, and in the test data with multiple, it is likely to be treated as outlier, different from the distribution for which the NN was trained. Should I delete my answer?
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-07T13:53:43.023Z
Answers.Text: Your answer contains useful information, but maybe you should also add some information that more directly addresses the question. So, I wouldn't delete your answer, if you plan to include this info.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-07T08:58:50.603Z
Answers.LastActivityDate: 2021-02-07T14:11:34.103Z
Answers.LastEditDate: 2021-02-07T14:11:34.103Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>I'm working on semantic segmentation tasks in the medical space using the U-Net. Let's say that I train a U-Net model on medical images with the goal of segmenting out, say, ligaments, from a medical image. If I train that model on images that contain just a single labelled ligament, it will be able to segment out single ligaments pretty well, I assume. If I present it with an image with multiple ligaments, should it also be able to segment the multiple ligaments well too?</p>
  <p>Based on my understanding, semantic segmentation is just pixel-wise classification. As a result, shouldn't the number of the objects in the image not be relevant since it's only looking at individual pixels? So, as long as a pixel matches that of a ligament, it should be able to segment it equally right?</p>
  <p>Or am I misunderstanding some piece?</p>
  <p>Basically, if I train a U-Net on images with just single ligaments, will it also be able to segment images with multiple ligaments equally as well based on my logic above?</p>
  
Tags: <computer-vision><image-segmentation><u-net><instance-segmentation><semantic-segmentation>
Title: If I trained a model to perform semantic segmentation on images with only one object, would it also work on images with multiple objects?
UserDisplayName: cmed123
ThreadText: # If I trained a model to perform semantic segmentation on images with only one object, would it also work on images with multiple objects?
  
  (Asked by: username_0 on 2020-01-12)
  
  I'm working on semantic segmentation tasks in the medical space using the U-Net. Let's say that I train a U-Net model on medical images with the goal of segmenting out, say, ligaments, from a medical image. If I train that model on images that contain just a single labelled ligament, it will be able to segment out single ligaments pretty well, I assume. If I present it with an image with multiple ligaments, should it also be able to segment the multiple ligaments well too?
  
  Based on my understanding, semantic segmentation is just pixel-wise classification. As a result, shouldn't the number of the objects in the image not be relevant since it's only looking at individual pixels? So, as long as a pixel matches that of a ligament, it should be able to segment it equally right?
  
  Or am I misunderstanding some piece?
  
  Basically, if I train a U-Net on images with just single ligaments, will it also be able to segment images with multiple ligaments equally as well based on my logic above?
  
  **Tags:** computer-vision, image-segmentation, u-net, instance-segmentation, semantic-segmentation
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2020-01-14. Score: 0
  
  Without experimental evidence to back me up, I can not answer this with 100% confidence. However, I am fairly certain that this will cause issues depending on the model.
  
  U-net is essentially an auto-encoder, and due to the fact that it is all just one big neural network, it is likely it will learn the easiest pattern (as all NN do), and that is to find one single instance of an object and shade that region.
  
  Now why does this depend on the model? Well let's say you are using something slightly different, where region proposals are generated by a determinisitc algorithm we've predefined, then these regions are run through a CNN to segment them. In this case, as each region is without context of the entire image, the difference between 2 objects in an image and 1 is indistinguishable to the network (as regions may overlap), and as such, only using images with 1 object will not pose any problems (there is a name for early models like these, though it escapes me).
  
  So assuming I am correct, what should you do? The models that use a deterministic algorithm for region proposals are slow and old, so I wouldn't suggest that. Instead, I would think that you should first do some testing, to see if it actually does cause issues. Assuming it does, a good option could be to tamper with the training data and separate segments by a few pixels to sort of "force" multiple objects into existence.
  
  Regardless of such, I would still suggest using U-net. Fixing this issue (if it does arrise) should be relatively easy to do, so there's little to lose by using U-net and just trying the training.
  
  
  ---
  
  ### Answer by username_2 on 2021-03-09. Score: 0
  
  I'm not aware of a model that could generalize from single segment to multi segment without nontrivial surgery, augmentation, and retraining, but I have had some success doing this in cases where the regions to segment are non-overlapping: After finding the first segment, edit the input image to distort or obscure the segmented region and run it through again, repeating until it stops finding new segments.
  
  
  ---
  
  ### Answer by username_3 on 2021-02-07. Score: 0
  
  In the problem of `Semantic Segmentation` one solves the problem of investigating, whether a particular pixel on the image belongs to a certain class. For example, on the photo taken from the street one may be interested , whether this pixel belongs to the road, traffic light, sign, people or background of no interest.
  
  Supeficially, in this setting it doesn't matter, whether there is a single instance of an object or multiple, and the thing, which is more important is the abundance of a particular class on an image.
  
  However, `CNN` adopt to a particular patterns, pertinent to the distribution, from which the data comes from. It can be the case, that a network has captured some property and tooks a pattern, which is relevant only for one instance case. Images with multiple instances of class of interest are like outliers in the data, and may be treated due to this incorrecty.
  
  I am not sure, whether it makes sense in yout particular problem, but maybe it is worth performing a `mosaic` augmentation <https://towardsdatascience.com/data-augmentation-in-yolov4-c16bd22b2617>. So that you will have for some samples of the augmented data a multiinstance case.
  
  > **Comments:**
  > **username_4**: I think you misunderstood the question. The question is not about the difference between semantic and instance segmentation. The question is whether the model would generalize and be able to segment images with a different number of objects than the number of objects that can be found in the training data. I think the OP is not interested in knowing whether the model would be able to differentiate the instances of the same type of object (i.e. they were not asking about instance segmentation). So, I don't think that this answer really addresses the question.  
  > **username_3**: @username_4 I mean that superficially due to the absence of notion of single or multiple instances of some class the task is the same. However, if in the training data there are instances with only single intstance, and in the test data with multiple, it is likely to be treated as outlier, different from the distribution for which the NN was trained. Should I delete my answer?  
  > **username_4**: Your answer contains useful information, but maybe you should also add some information that more directly addresses the question. So, I wouldn't delete your answer, if you plan to include this info.  
  
  ---
  
  