Deep Visual Analogy-Making
In addition to identifying the content within a single image, relating images and generating related images are critical tasks for image understanding. Recently, deep convolutional networks have yielded breakthroughs in predicting image labels, annotations and captions, but have only just begun to be used for generating high-quality images. In this paper we develop a novel deep network trained end-to-end to perform visual analogy making, which is the task of transforming a query image according to an example pair of related images. Solving this problem requires both accurately recognizing a visual relationship and generating a transformed query image accordingly. Inspired by recent advances in language modeling, we propose to solve visual analogies by learning to map images to a neural embedding in which analogical reasoning is simple, such as by vector subtraction and addition. In experiments, our model effectively models visual analogies on several datasets: 2D shapes, animated video game sprites, and 3D car models.
This paper proposes a method to find analogical images using given query images. Specifically, given four images a, b, c, and d, we would like to find a transformed version of c such that it is similar to the transformation from a to b when applied to d. This is an interesting problem and should be of interest to image recognition community in general because if images could be represented in a space (vectors in the proposed method), then such translation invariant representations could simplify the task of recognizing images thereby improving recognition accuracies.Although using analogical methods for image recognition is a relatively new topic in the visual domain, it has a well established history in NLP. For example, SAT word analogy questions have been used to evaluate the performance of analogy detection methods. See @article{Turney_CL,   Author = {P.D. Turney},   Journal = {Computational Linguistics},   Number = {3},   Pages = {379--416},   Title = {Similarity of semantic relations},   Volume = {32},   Year = {2006}} for an overview. Both supervised and unsupervised methods have been proposed for this purpose. Some background description of such prior work on analogy detection will give a good context to the current paper.Moreover, some work has been done in the visual domain studying embeddings that preserve analogical properties in images. @inproceedings{Hwang:ICML:2013,   Author = {Sung Ju Hwang and Kristen Grauman and Fei Sha},   Booktitle = {ICML'13},   Title = {Analogy-preserving Semantic Embedding for Visual Object Categorization},   Year = {2013}}Sec 3.1 argues that sampling from co-occurring contexts results in word embeddings that demonstrate analogical properties when the vector difference is considered. However, this claim is not clear to me. Why would predicting word co-occurrence result in word representations that demonstrate analogies? It is true empirically that this is the case but why predicting co-occurrences should help?In the proposed method this is by design. In other words, the objective functions are designed in such a way to minimize the loss between the target image d and its construction via a transform on a, b, and c. There is a recent work that learns semantic representations for words by predicting analogies directly in similar spirit to the proposed method. Embedding semantic relations into Word Representations IJCAI 2015 arxiv paper.In Algo 1, once you create N candidate images how would you select the best one for d? Do you evaluate the objective function for each candidate and select the best one? A related question is how to decide N? Does it depend on the difference between a and b?One weakness of the evaluation section is that all evaluations are limited to different baselines/settings within the proposed method. It would have been good if there was some comparison against prior work.