Answers.Body: <p>For something like this you may just want to compare the image features directly, rather than trying to fanangle your data into a class based segmentation model.  There are a number of very fast feature detection algorithms, like SIFT, SURF, ORB, and BRIEF, that don't necessitate the use of a neural network at all.</p>
  <p><a href="https://docs.opencv.org/3.4/dc/dc3/tutorial_py_matcher.html" rel="nofollow noreferrer">OpenCV - Feature Matching</a></p>
  <p>The features locked away in the hidden layer of a CNN are gold for this sort of thing as well, and chopping the head off of a pretrained neural network to get at those conv layers, and then using those in a manner similar to SIFT features, may improve accuracy while accomplishing the same end.</p>
  <p><a href="https://openaccess.thecvf.com/content_cvpr_2017/papers/Ufer_Deep_Semantic_Feature_CVPR_2017_paper.pdf" rel="nofollow noreferrer">Paper - Deep Semantic Feature Matching</a></p>
  <p>Either way, you are going to be doing some kind of similarity search here, rather than a classification task.  You want to use methods that see if the important bits of image A exist in image B, and any attempt to do that via classification models would create so many classes as to be unusable.  Any method used to accomplish this task is going to have to rely on similarity of image features in one way or another.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T22:27:40.553Z
Answers.LastActivityDate: 2024-01-02T22:27:40.553Z
Answers.UserDisplayName: Jeremiah
Answers.Body: <p>You can formulate your problem as a <strong>template matching (TM)</strong> one. This task is similar to object detection, but you don't need to label anything. There are 2 images: the template (in your case, the template is the image of a product) and the image where you want to search the template (the image with the shelf of products). There are different TM algorithms. For example, <a href="https://docs.opencv.org/4.x/d4/dc6/tutorial_py_template_matching.html" rel="nofollow noreferrer">OpenCV</a> offers a simple one with different similarity measures.</p>
  <p>From my limited experience with TM, it works well only if the objects in both the template and the other image are in a plane parallel to image plane. This is more or less your case (at least, in the images you're showing us), so I expect that TM will work reasonably well with those 2 specific images, but you should first resize the first image so that there are no padding around the object (the chocolate bar) - in a way, this is like labeling. If this feasible, then you should try TM.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-12T19:20:27.250Z
Answers.Text: I am given a **Hint: Use Visual Similarity**. Maybe I am wrong but I believe TM isn't of much use here because time spent resizing all the product images is as good as labeling, so please usher me to a path like where I have a pre-trained model on a similar dataset and now I would like to freeze some layers and fine-tune the model on a new dataset which comes with a problem that it does not have labels or annotations. So how should I proceed from there? Note : You can assume I have no experience with self-supervising the pre-trained model, or maybe something better that you can suggest?
Answers.UserDisplayName: Aarush Aggarwal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-12T20:29:00.967Z
Answers.Text: I don't know who gave you that hint, but template matching uses some kind of "visual similarity". There might be a tool that attempts to automatically resize the image to fit the object in it. So, resizing might be less laborious than labelling and maybe you don't have to do a lot manual work. Moreover, you probably don't have to be super precise when resizing. You can still leave some padding/spacing around the object and it might still work. Having said that, it might be possible to use a pre-trained model to do your task, but I don't know which one - I am not an expert in this topic.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-12T15:46:21.963Z
Answers.LastActivityDate: 2022-04-12T16:00:49.083Z
Answers.LastEditDate: 2022-04-12T16:00:49.083Z
Answers.UserDisplayName: nbro
Body: <p><strong>Problem Statement</strong>:</p>
  <p>I am given 2 sets of images. All the images in both sets are without annotations and labels.</p>
  <p>First set : a set of images of the grocery store shelves (captured in the grocery stores).</p>
  <p>Second set: a set of close-up images of the products kept on those store shelves.</p>
  <p><strong>What I am trying to achieve:</strong></p>
  <p><strong>Goal:</strong> For every product image, I want to find the location of that product in all shelf images in which it appears.</p>
  <p>For example,</p>
  <p><strong>Input</strong> <strong>Product image</strong>
  <a href="https://i.sstatic.net/pG3WK.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pG3WK.jpg" alt="Product image" /></a></p>
  <p><strong>Output</strong> <strong>Corresponding</strong> <strong>Shelf image</strong>
  <a href="https://i.sstatic.net/ahxZK.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ahxZK.jpg" alt="Shelf image" /></a></p>
  <p><strong>My approach:</strong></p>
  <ol>
  <li>For each product image, first find all the shelf image(s) which contain that product.</li>
  <li>Then predict a bounding box by finding the location of the product in the shelf image.</li>
  </ol>
  <p>I am using YOLOv5 for this task but I am not sure how should I start off given that I have to do it without annotations or labels.</p>
  <p>I have come across terms like Zero-shot learning, self-supervised Object Detection, etc. but I haven't been able to figure out their use case as a starting point.</p>
  <p>There's a <a href="https://stackoverflow.com/questions/66353237/object-detection-without-labels-annotation">similar question</a> asked on StackOverflow but I am not sure the answer to it solves the problem.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-12T14:20:37.243Z
Comments.Text: Can you please put your **specific question** in the title? "Object Detection without annotations and labels" is not a question and it's also not very specific.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-12T14:43:08.887Z
Comments.Text: @nbro Done! If I may ask, does that make it more clear now? For a detailed description of the problem, please read the explanation down there that I have provided along with the newly edited title.
Comments.UserDisplayName: Aarush Aggarwal
Tags: <computer-vision><image-recognition><object-detection><object-recognition><yolo>
Title: For an image (of any object), how to find its location in the other image(s) which contains it, given there are no labels or annotations for any image
UserDisplayName: Aarush Aggarwal
ThreadText: # For an image (of any object), how to find its location in the other image(s) which contains it, given there are no labels or annotations for any image
  
  (Asked by: username_0 on 2022-04-09)
  
  **Problem Statement**:
  
  I am given 2 sets of images. All the images in both sets are without annotations and labels.
  
  First set : a set of images of the grocery store shelves (captured in the grocery stores).
  
  Second set: a set of close-up images of the products kept on those store shelves.
  
  **What I am trying to achieve:**
  
  **Goal:** For every product image, I want to find the location of that product in all shelf images in which it appears.
  
  For example,
  
  **Input** **Product image**
  [![Product image](https://i.sstatic.net/pG3WK.jpg)](https://i.sstatic.net/pG3WK.jpg)
  
  **Output** **Corresponding** **Shelf image**
  [![Shelf image](https://i.sstatic.net/ahxZK.jpg)](https://i.sstatic.net/ahxZK.jpg)
  
  **My approach:**
  
  1. For each product image, first find all the shelf image(s) which contain that product.
  2. Then predict a bounding box by finding the location of the product in the shelf image.
  
  I am using YOLOv5 for this task but I am not sure how should I start off given that I have to do it without annotations or labels.
  
  I have come across terms like Zero-shot learning, self-supervised Object Detection, etc. but I haven't been able to figure out their use case as a starting point.
  
  There's a [similar question](https://stackoverflow.com/questions/66353237/object-detection-without-labels-annotation) asked on StackOverflow but I am not sure the answer to it solves the problem.
  
  **Tags:** computer-vision, image-recognition, object-detection, object-recognition, yolo
  > **Comments:**
  > **username_1**: Can you please put your **specific question** in the title? "Object Detection without annotations and labels" is not a question and it's also not very specific.  
  > **username_0**: @nbro Done! If I may ask, does that make it more clear now? For a detailed description of the problem, please read the explanation down there that I have provided along with the newly edited title.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-01-02. Score: 0
  
  For something like this you may just want to compare the image features directly, rather than trying to fanangle your data into a class based segmentation model. There are a number of very fast feature detection algorithms, like SIFT, SURF, ORB, and BRIEF, that don't necessitate the use of a neural network at all.
  
  [OpenCV - Feature Matching](https://docs.opencv.org/3.4/dc/dc3/tutorial_py_matcher.html)
  
  The features locked away in the hidden layer of a CNN are gold for this sort of thing as well, and chopping the head off of a pretrained neural network to get at those conv layers, and then using those in a manner similar to SIFT features, may improve accuracy while accomplishing the same end.
  
  [Paper - Deep Semantic Feature Matching](https://openaccess.thecvf.com/content_cvpr_2017/papers/Ufer_Deep_Semantic_Feature_CVPR_2017_paper.pdf)
  
  Either way, you are going to be doing some kind of similarity search here, rather than a classification task. You want to use methods that see if the important bits of image A exist in image B, and any attempt to do that via classification models would create so many classes as to be unusable. Any method used to accomplish this task is going to have to rely on similarity of image features in one way or another.
  
  
  ---
  
  ### Answer by username_1 on 2022-04-12. Score: 0
  
  You can formulate your problem as a **template matching (TM)** one. This task is similar to object detection, but you don't need to label anything. There are 2 images: the template (in your case, the template is the image of a product) and the image where you want to search the template (the image with the shelf of products). There are different TM algorithms. For example, [OpenCV](https://docs.opencv.org/4.x/d4/dc6/tutorial_py_template_matching.html) offers a simple one with different similarity measures.
  
  From my limited experience with TM, it works well only if the objects in both the template and the other image are in a plane parallel to image plane. This is more or less your case (at least, in the images you're showing us), so I expect that TM will work reasonably well with those 2 specific images, but you should first resize the first image so that there are no padding around the object (the chocolate bar) - in a way, this is like labeling. If this feasible, then you should try TM.
  
  > **Comments:**
  > **username_0**: I am given a **Hint: Use Visual Similarity**. Maybe I am wrong but I believe TM isn't of much use here because time spent resizing all the product images is as good as labeling, so please usher me to a path like where I have a pre-trained model on a similar dataset and now I would like to freeze some layers and fine-tune the model on a new dataset which comes with a problem that it does not have labels or annotations. So how should I proceed from there? Note : You can assume I have no experience with self-supervising the pre-trained model, or maybe something better that you can suggest?  
  > **username_1**: I don't know who gave you that hint, but template matching uses some kind of "visual similarity". There might be a tool that attempts to automatically resize the image to fit the object in it. So, resizing might be less laborious than labelling and maybe you don't have to do a lot manual work. Moreover, you probably don't have to be super precise when resizing. You can still leave some padding/spacing around the object and it might still work. Having said that, it might be possible to use a pre-trained model to do your task, but I don't know which one - I am not an expert in this topic.  
  
  ---
  
  