VISALOGY: Answering Visual Analogy Questions
In this paper, we study the problem of answering visual analogy questions. These questions take the form of image A is to image B as image C is to what. Answering these questions entails discovering the mapping from image A to image B and then extending the mapping to image C and searching for the image D such that the relation from A to B holds for C to D. We pose this problem as learning an embedding that encourages pairs of analogous images with similar transformations to be close together using convolutional neural networks with a quadruple Siamese architecture. We introduce a dataset of visual analogy questions in natural images, and show first results of its kind on solving analogy questions on natural images.
The paper presents a model for visual analogy retrieval. For a query image one wants to find a second image which relates to the query in the same way as the relationship between a given pair of images. More precisely, an offset pair (i1, i2) defines the analogy. For a query i3 an image i4 is to be found such that i3 relates to i4 in the same way as i1 relates to i2.The model is formulated as an embedder, which embeds all four images ik -> xk (k = 1, ..., 4) such that (x4 - x3) = (x2 - x1). The embedder is a CNN based on AlexNet. The loss is a hinge loss. In addition, the authors present a dataset of visual analogy. The paper is well written and presents a well defined model which is empirically evaluate in two datasets. My main concern with this work is that the presented problem might be a bit contrived. The authors define an analogy as relationship between images defined by different properties of these images. The philosophy of analogies being a pair of two properties is incorporated in both of their datasets.So if an analogy is defined as pair of properties, I natural baseline would be to set up a classification problem for these properties. Then if analogy is defined by the properties of two images, then one can apply the trained classifier to estimate these properties. A second classifier would determine the class of the query image. Then the analogous image would be one of the class of the query and the missing of the two properties. Such a baseline is based on two classifiers and would be simple to set up.