Answers.Body: <p>First and foremost, I have to say that this could (and likely will) be a <strong>very hard task</strong>. Neural networks (<strong>NNs</strong>) have excelled at computer vision tasks identifying everything from textures to complex objects but what you are trying to do goes beyond that. We (humans) identify trash using the context as much as the object. An object on a table and the same object in a trash bin can <strong>look identical</strong> but the context tells us which one is trash. Also, <strong>anything</strong> can be trash. Trash is not an object, its a state.</p>
  
  <p>Having said all that, it sounds like an interesting project and it would be a very useful model so I'll do my best to explain how you would go about trying this. As per your comment, you have outlined the two steps you need.</p>
  
  <ol>
  <li>Identify and extract the trash object from the image.</li>
  <li>Classify the type of waste</li>
  </ol>
  
  <p>This could be accomplished using a single model but I'll explain how to do it with two models to make clear what is being done.</p>
  
  <h2>Identify and extract trash</h2>
  
  <p>The objective of the first model is to identify and extract the region of interest (the part of the image containing the trash). This is an <a href="https://www.analyticsvidhya.com/blog/2019/04/introduction-image-segmentation-techniques-python/" rel="nofollow noreferrer">image segmentation</a> task typically accomplished using an <a href="https://arxiv.org/abs/1311.2524" rel="nofollow noreferrer">R-CNN</a> - some guides explaining how they work can be found <a href="https://towardsdatascience.com/computer-vision-instance-segmentation-with-mask-r-cnn-7983502fcad1" rel="nofollow noreferrer">here</a>, <a href="https://towardsdatascience.com/r-cnn-3a9beddfd55a" rel="nofollow noreferrer">here</a> or <a href="https://cv-tricks.com/object-detection/faster-r-cnn-yolo-ssd/" rel="nofollow noreferrer">here</a>. These methods use supervised learning which requires masks delineating the object of interest to be used as the <em>ground truth</em>. A mask is a binary image the same size as your input image where each pixel in the image representing your positive class (trash) is set to 1 and all others are set to 0.</p>
  
  <p>The output of your Canny/Watershed algorithms could be used as the masks to train the model, <strong>however</strong> your model will only ever be as good as your masks. Therefore, you might as well use your Canny/Watershed algorithms to accomplish task <strong>1</strong>. If the masks generated by your algorithm are not of sufficient accuracy you will need to find another way to generate your masks - maybe even doing it manually. </p>
  
  <p>An approximate rule of thumb for object detection with NNs is that you need 1,000 representative images per class, where class implies a specific object. In this case, unless you are attempting to identify very specific items of trash, you would likely need 100s of thousands of images to obtain a high degree of accuracy.</p>
  
  <h2>Classify the type of waste</h2>
  
  <p>This task should be easier, making the assumption that the segmented image from the first step is always trash. By assigning a label to each segmented image (paper, metal, glass, cardboard, etc) it becomes a normal <strong>multi-class classification</strong> task which are well documented online with lots of explainations and tutorials.</p>
  
  <h2>Single model</h2>
  
  <p>These tasks could be combined into a single model by modifying the masks in step 1. Instead of using a single binary mask, you could create a multichannel mask of shape <code>m x n x o</code> where <code>m x n</code> is your input image size and <code>o</code> is the number of kinds of waste you are attempting to identify. Each channel is a binary mask for a given type of waste. Therefore, it becomes a matter of not just identifying and segmenting trash but of segmenting each type of trash separately. Needless to say the complexity of the model would be a lot higher and so would the process to create the masks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-05T19:52:56.323Z
Answers.LastActivityDate: 2020-05-05T19:52:56.323Z
Answers.UserDisplayName: tynowell
Body: <p>I have a very silly problem in hand. I have implemented 2 methods which give me the mask to separate the objects from the background. What I get from one method is the object encapsulated in the red Contour or boundary and the other one makes the background Red. </p>
  
  <p>I am using Keras to classify Trash. I wanted to use the output or the <code>EXTRACTED</code> object as an input to the CNN model. Now I do not see any difference in the images and output. All there is extra boundary around the object and I fail to understand how it can help my model.</p>
  
  <p>I could add an extra <code>alpha</code> channel to second method to make the background transparent but Keras' <code>ImageDataGenerator</code> do not work with <code>RGBA</code> images. What shoul dI do to improve the model?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-04T21:03:27.007Z
Comments.Text: By 'classify trash', do you mean a binary classifier which gives a boolean output for a given image (trash in image: true/false), or are you attempting to segment the trash in the image (a binary mask indicating which pixels represent trash in the image)
Comments.UserDisplayName: tynowell
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-05T07:09:45.013Z
Comments.Text: I want to extract the `object` from the image of garbage and then predict what kind of waste it is (paper,metal,glass,cardboard etc) using a Keras model.  There is just one object per image.
Comments.UserDisplayName: Deshwal
Tags: <convolutional-neural-networks><computer-vision><image-recognition><keras><image-segmentation>
Title: How to use 'Canny/Watershed' algorithm's output as an input for Image Classification Model
UserDisplayName: Deshwal
ThreadText: # How to use 'Canny/Watershed' algorithm's output as an input for Image Classification Model
  
  (Asked by: username_0 on 2020-05-02)
  
  I have a very silly problem in hand. I have implemented 2 methods which give me the mask to separate the objects from the background. What I get from one method is the object encapsulated in the red Contour or boundary and the other one makes the background Red.
  
  I am using Keras to classify Trash. I wanted to use the output or the `EXTRACTED` object as an input to the CNN model. Now I do not see any difference in the images and output. All there is extra boundary around the object and I fail to understand how it can help my model.
  
  I could add an extra `alpha` channel to second method to make the background transparent but Keras' `ImageDataGenerator` do not work with `RGBA` images. What shoul dI do to improve the model?
  
  **Tags:** convolutional-neural-networks, computer-vision, image-recognition, keras, image-segmentation
  
  > **Comments:**
  > **username_1**: By 'classify trash', do you mean a binary classifier which gives a boolean output for a given image (trash in image: true/false), or are you attempting to segment the trash in the image (a binary mask indicating which pixels represent trash in the image)  
  > **username_0**: I want to extract the `object` from the image of garbage and then predict what kind of waste it is (paper,metal,glass,cardboard etc) using a Keras model.  There is just one object per image.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-05. Score: 1
  
  First and foremost, I have to say that this could (and likely will) be a **very hard task**. Neural networks (**NNs**) have excelled at computer vision tasks identifying everything from textures to complex objects but what you are trying to do goes beyond that. We (humans) identify trash using the context as much as the object. An object on a table and the same object in a trash bin can **look identical** but the context tells us which one is trash. Also, **anything** can be trash. Trash is not an object, its a state.
  
  Having said all that, it sounds like an interesting project and it would be a very useful model so I'll do my best to explain how you would go about trying this. As per your comment, you have outlined the two steps you need.
  
  1. Identify and extract the trash object from the image.
  2. Classify the type of waste
  
  This could be accomplished using a single model but I'll explain how to do it with two models to make clear what is being done.
  
  ## Identify and extract trash
  
  The objective of the first model is to identify and extract the region of interest (the part of the image containing the trash). This is an [image segmentation](https://www.analyticsvidhya.com/blog/2019/04/introduction-image-segmentation-techniques-python/) task typically accomplished using an [R-CNN](https://arxiv.org/abs/1311.2524) - some guides explaining how they work can be found [here](https://towardsdatascience.com/computer-vision-instance-segmentation-with-mask-r-cnn-7983502fcad1), [here](https://towardsdatascience.com/r-cnn-3a9beddfd55a) or [here](https://cv-tricks.com/object-detection/faster-r-cnn-yolo-ssd/). These methods use supervised learning which requires masks delineating the object of interest to be used as the *ground truth*. A mask is a binary image the same size as your input image where each pixel in the image representing your positive class (trash) is set to 1 and all others are set to 0.
  
  The output of your Canny/Watershed algorithms could be used as the masks to train the model, **however** your model will only ever be as good as your masks. Therefore, you might as well use your Canny/Watershed algorithms to accomplish task **1**. If the masks generated by your algorithm are not of sufficient accuracy you will need to find another way to generate your masks - maybe even doing it manually.
  
  An approximate rule of thumb for object detection with NNs is that you need 1,000 representative images per class, where class implies a specific object. In this case, unless you are attempting to identify very specific items of trash, you would likely need 100s of thousands of images to obtain a high degree of accuracy.
  
  ## Classify the type of waste
  
  This task should be easier, making the assumption that the segmented image from the first step is always trash. By assigning a label to each segmented image (paper, metal, glass, cardboard, etc) it becomes a normal **multi-class classification** task which are well documented online with lots of explainations and tutorials.
  
  ## Single model
  
  These tasks could be combined into a single model by modifying the masks in step 1. Instead of using a single binary mask, you could create a multichannel mask of shape `m x n x o` where `m x n` is your input image size and `o` is the number of kinds of waste you are attempting to identify. Each channel is a binary mask for a given type of waste. Therefore, it becomes a matter of not just identifying and segmenting trash but of segmenting each type of trash separately. Needless to say the complexity of the model would be a lot higher and so would the process to create the masks.
  
  
  ---
  
  