Answers.Body: <p>From your examples I assume you presuppose image recognition in the sense that you don't compare the actual images, but the descriptions of what the images contain.</p>
  
  <p>For comparing images there are various algorithms working on the visual similarity. This can sometimes lead to interesting results, as you probably have seen images on the internet like "<a href="https://starecat.com/content/wp-content/uploads/dog-face-or-a-muffin-guess.jpg" rel="nofollow noreferrer">dog or muffin</a>". A purely visual approach would find this hard to do.</p>
  
  <p>However, if you do have the description of the image (as there are ways of getting captions from images), then it would just be a text comparison between three sentences: the one describing your query image, and those describing your images X and Y. There are ways of getting at the semantic similarity of sentences. The simplest way (from your examples) would be to look at the overlap in words: [yellow, car]/Q, [red, car]/X, and [yellow, car]/Y obviously has the largest overlap with Q and Y. This is rather simplistic, but it is what your examples suggest you are dealing with.</p>
  
  <p>But, getting a proper description from an image is a hard task, of the calibre that Google and Instagram are still working on; they have large amounts of training data and huge resources they can throw at the problem. So, unless that is your starting point, it will not be easy to achieve.</p>
  
  <p>Overall I do not think it is currently possible to solve this problem with high accuracy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-21T15:11:07.650Z
Answers.Text: I would go further and say that it is not even possible to define the problem with high accuracy, although I like the approach of comparing open-ended text annotations, because in theory these will emulate the subjectivity of comparing photos done by people (at least in a given context, such as how they would be labelled when uploaded to the internet).
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-21T21:57:04.203Z
Answers.Text: Thanks for the idea. In fact I do have access to some textual metadata, but unfortunately that would only help me solve the easy cases, e.g. "a car is probably more similar to another car than to a dog", however for the trickier cases textual metadata falls short, which makes me believe I need to resort to some kind of fine-grained visual features to break ties. In fact my goal is to learn an embedding with a Neural Network using a Triplet Loss, but the crucial problem lies in generating accurate triplet samples, including both easy and tricky cases. Hence my question.
Answers.UserDisplayName: Pablo Messina
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-21T22:21:39.580Z
Answers.Text: One idea I have in mind is to extract a ton of features using different techniques (pre-trained CNNs, handcrafted features, etc.) and somehow figure out a hybrid score that combines the "opinions" of all the individual features. A sort of "mixture of experts". This hybrid score would let me judge which image is the most similar to the query image Q. But I have no idea of how to define such a hybrid score.
Answers.UserDisplayName: Pablo Messina
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-21T13:51:06.630Z
Answers.LastActivityDate: 2018-05-21T13:51:06.630Z
Answers.UserDisplayName: Oliver Mason
Body: <p>Given a query image Q and two other images X and Y (you can assume they have more or less the same resolutions if that simplifies the problem), which algorithm would perform extremely well at determining which image between X and Y is most similar to Q, even when the differences are rather subtle? For example, a trivial case would be:</p>
  
  <ul>
  <li>Q = image of mountains, X = image of mountains, Y = image of dogs, therefore it is clear that sim(Q,X) > sim(Q,Y).</li>
  </ul>
  
  <p>However, examples of trickier cases would be: </p>
  
  <ul>
  <li>Q = image of a yellow car, X = image of a red car, Y = image of a yellow car, therefore sim(Q,Y) > sim(Q,X) (assuming the car shapes are more or less the same).</li>
  <li>Q = image of a man standing up in the middle with a black background, X = image of another man standing up in the middle with a black background, Y = image of a woman standing up in the middle with a black background, therefore sim(Q,X) > sim(Q,Y).</li>
  </ul>
  
  <p>Which algorithm (or combination of algorithms) would be robust enough to handle even the tricky cases with very high accuracy?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-05-20T19:48:25.573Z
Comments.Text: Are you commenting in the wrong question by any chance? I don't understand why you are talking about time, simultaneity, past, present and future.
Comments.UserDisplayName: Pablo Messina
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-05-20T20:49:10.220Z
Comments.Text: @Bobs, I really think you are just trolling. Your comments have nothing to do with my original question. Please stay on-topic.
Comments.UserDisplayName: Pablo Messina
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-05-21T15:07:32.240Z
Comments.Text: A simpler way of putting this: The OP has to define "similarity" by some kind of measure. There is no single correct, objective measure. The section labelled "trickier cases" gets closer to it. Consider the two men and woman against plain background. Which is more important: 1) pose of person (is one man standing with back facing camera, woman and man both facing camera) 2) colour or nature of background 3) colour of hair/skin/eye, clothing or some cultural similarity between the people  . . . etc, etc. There is no "true" answer, the goal can be highly subjective, depending on the data set
Comments.UserDisplayName: Neil Slater
Tags: <algorithm><computer-vision>
Title: Given a query image Q and two other images X and Y, how to determine which one is most similar to Q?
UserDisplayName: Pablo Messina
ThreadText: # Given a query image Q and two other images X and Y, how to determine which one is most similar to Q?
  
  (Asked by: username_0 on 2018-05-20)
  
  Given a query image Q and two other images X and Y (you can assume they have more or less the same resolutions if that simplifies the problem), which algorithm would perform extremely well at determining which image between X and Y is most similar to Q, even when the differences are rather subtle? For example, a trivial case would be:
  
  * Q = image of mountains, X = image of mountains, Y = image of dogs, therefore it is clear that sim(Q,X) > sim(Q,Y).
  
  However, examples of trickier cases would be:
  
  * Q = image of a yellow car, X = image of a red car, Y = image of a yellow car, therefore sim(Q,Y) > sim(Q,X) (assuming the car shapes are more or less the same).
  * Q = image of a man standing up in the middle with a black background, X = image of another man standing up in the middle with a black background, Y = image of a woman standing up in the middle with a black background, therefore sim(Q,X) > sim(Q,Y).
  
  Which algorithm (or combination of algorithms) would be robust enough to handle even the tricky cases with very high accuracy?
  
  **Tags:** algorithm, computer-vision
  
  > **Comments:**
  > **username_0**: Are you commenting in the wrong question by any chance? I don't understand why you are talking about time, simultaneity, past, present and future.  
  > **username_0**: @Bobs, I really think you are just trolling. Your comments have nothing to do with my original question. Please stay on-topic.  
  > **username_1**: A simpler way of putting this: The OP has to define "similarity" by some kind of measure. There is no single correct, objective measure. The section labelled "trickier cases" gets closer to it. Consider the two men and woman against plain background. Which is more important: 1) pose of person (is one man standing with back facing camera, woman and man both facing camera) 2) colour or nature of background 3) colour of hair/skin/eye, clothing or some cultural similarity between the people  . . . etc, etc. There is no "true" answer, the goal can be highly subjective, depending on the data set  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-05-21. Score: 2
  
  From your examples I assume you presuppose image recognition in the sense that you don't compare the actual images, but the descriptions of what the images contain.
  
  For comparing images there are various algorithms working on the visual similarity. This can sometimes lead to interesting results, as you probably have seen images on the internet like "[dog or muffin](https://starecat.com/content/wp-content/uploads/dog-face-or-a-muffin-guess.jpg)". A purely visual approach would find this hard to do.
  
  However, if you do have the description of the image (as there are ways of getting captions from images), then it would just be a text comparison between three sentences: the one describing your query image, and those describing your images X and Y. There are ways of getting at the semantic similarity of sentences. The simplest way (from your examples) would be to look at the overlap in words: [yellow, car]/Q, [red, car]/X, and [yellow, car]/Y obviously has the largest overlap with Q and Y. This is rather simplistic, but it is what your examples suggest you are dealing with.
  
  But, getting a proper description from an image is a hard task, of the calibre that Google and Instagram are still working on; they have large amounts of training data and huge resources they can throw at the problem. So, unless that is your starting point, it will not be easy to achieve.
  
  Overall I do not think it is currently possible to solve this problem with high accuracy.
  
  > **Comments:**
  > **username_1**: I would go further and say that it is not even possible to define the problem with high accuracy, although I like the approach of comparing open-ended text annotations, because in theory these will emulate the subjectivity of comparing photos done by people (at least in a given context, such as how they would be labelled when uploaded to the internet).  
  > **username_0**: Thanks for the idea. In fact I do have access to some textual metadata, but unfortunately that would only help me solve the easy cases, e.g. "a car is probably more similar to another car than to a dog", however for the trickier cases textual metadata falls short, which makes me believe I need to resort to some kind of fine-grained visual features to break ties. In fact my goal is to learn an embedding with a Neural Network using a Triplet Loss, but the crucial problem lies in generating accurate triplet samples, including both easy and tricky cases. Hence my question.  
  > **username_0**: One idea I have in mind is to extract a ton of features using different techniques (pre-trained CNNs, handcrafted features, etc.) and somehow figure out a hybrid score that combines the "opinions" of all the individual features. A sort of "mixture of experts". This hybrid score would let me judge which image is the most similar to the query image Q. But I have no idea of how to define such a hybrid score.  
  
  ---
  
  