Similarity Component Analysis
Measuring similarity is crucial to many learning tasks. To this end, metric learning has been the dominant paradigm. However, similarity is a richer and broader notion than what metrics entail. For example, similarity can arise from the process of aggregating the decisions of multiple latent components, where each latent component compares data in its own way by focusing on a different subset of features. In this paper, we propose Similarity Component Analysis (SCA), a probabilistic graphical model that discovers those latent components from data. In SCA, a latent component generates a local similarity value, computed with its own metric, independently of other components. The final similarity measure is then obtained by combining the local similarity values with a (noisy-)OR gate. We derive an EM-based algorithm for fitting the model parameters with similarity-annotated data from pairwise comparisons. We validate the SCA model on synthetic datasets where SCA discovers the ground-truth about the latent components. We also apply SCA to a multiway classification task and a link prediction task. For both tasks, SCA attains significantly better prediction accuracies than competing methods. Moreover, we show how SCA can be instrumental in exploratory analysis of data, where we gain insights about the data by examining patterns hidden in its latent components' local similarity values.
This paper proposes a graphical model for measuring 
      similarity, based on the assumption that two objects are similar because 
      some (instead of all) of their components are similar. Following this 
      design, they show empirically their model performs better than recent 
      metric learning approaches, when the number of components (K) is larger 
      than 1. Overall, I like this paper. The model design captures one 
      of the fundamental issues of similarity measures and differentiate it from 
      the metric learning framework. Despite the fact that it's a graphical 
      model, the approach still cleverly leverages some aspects of the metric 
      learning models. Specifically, the component similarity is essentially a 
      Mahalanobis distance (before using a sigmoid function to make it 
      probability). This design makes sure that the model would generalize the 
      metric learning approaches, which reflects in the experimental results as 
      well (when K=1). For some applications/tasks, when K increases, the 
      performance does improve, which demonstrates that this model fits those 
      cases better. On the other hand, since the model can be viewed as 
      an extension of the Mahalanobis learning, the scalability of the learning 
      process still seems restricted. For instance, PCA needs to be employed to 
      reduce the dimensionality of the raw data before applying this algorithm 
      (same applied to ITML and LMNN though). Special constraints on M_k need to 
      be enforced (SCA-DIAG, where the component similarity becomes a weighted 
      Euclidean distance [1]) to reduce the complexity. Such issues are not 
      critical, but it would be nice if the authors can further discuss them in 
      the paper. Another issue that I wish the authors could discuss further is 
      the sparseness of M_k. It is mentioned in the paper that M_k is sparse 
      naturally when learning the model. Why this happens is nevertheless not 
      very clear since there seems no special regularization on M_k. It would be 
      interesting if the authors could provide some insight. Other 
      comments: 1. The example of CENTAUR, MAN and HORSE is in fact the 
      canonical example to show that similarity does not need to enforce 
      triangular inequality, although the authors did not make this argument 
      explicit. See Figure 3 in [2]. 2. The component similarity design is 
      especially suitable for word similarity, where each word can have multiple 
      senses. For instance, JAGUAR and AUTO are similar. JAGUAR and PUMA are 
      also similar. However, these two pairs are similar because different 
      components (e.g., senses) of the word JAGUAR. One of the recent papers 
      that specifically addresses this issue is [3]. It would be interesting to 
      know if the proposed model can subsume their approach given. 3. 
      Perhaps it's out of the scope of this paper, but the model design bears 
      some resemblance to the Siamese neural network approach [4,5,6]. It should 
      be very straightforward to design a NN architecture with the component 
      similarity subnetworks, and thus interesting to see how it compares to the 
      graphical model approach. [1] Schultz and Joachims. Learning a 
      distance metric from relative comparisons. NIPS-2004. [2] D Lin. An 
      information-theoretic definition of similarity. ICML-1998. [3] 
      Reisinger and Mooney. Multi-Prototype Vector-Space Models of Word Meaning. 
      NAACL-2010. [4] Bromley, Bentz, Bottou, Guyon, LeCun, Moore, Sackinger 
      and Shah. Signature verification using a “Siamese” time delay neural 
      network. 1993. International Journal Pattern Recognition and Artificial 
      Intelligence, 7(4):669–688. [5] Yih, Toutanova, Platt and Meek. 
      Learning Discriminative Projections for Text Similarity Measures. 
      CoNLL-2011. [6] Weston, Bengio, and Usunier. Large scale image 
      annotation: Learning to rank with joint word-image embeddings. ECML-2010. 
      --- Thanks for your response to the 
      comments.