A Deep Architecture for Matching Short Texts
Many machine learning problems can be interpreted as learning for matching two types of objects (e.g., images and captions, users and products, queries and documents, etc.). The matching level of two objects is usually measured as the inner product in a certain feature space, while the modeling effort focuses on mapping of objects from the original space to the feature space. This schema, although proven successful on a range of matching tasks, is insufficient for capturing the rich structure in the matching process of more complicated objects. In this paper, we propose a new deep architecture to more effectively model the complicated matching relations between two objects from heterogeneous domains. More specifically, we apply this model to matching tasks in natural language, e.g., finding sensible responses for a tweet, or relevant answers to a given question. This new architecture naturally combines the localness and hierarchy intrinsic to the natural language problems, and therefore greatly improves upon the state-of-the-art models.
This paper proposes an approach to matching short 
      texts based on using topic modeling to identify the common word 
      co-occurrence patterns within pairs of matched documents. Topic 
      modeling is peformed several times using fewer and fewer topics, 
      producing a hierarchy over topics and thus word co-occurrence 
      patterns. The heirarchy is then used to define the structure of a 
      sparsely connected neural network, taking advantage of "semantic 
      locality". The model is then trained to score correct matches above 
      incorrect ones. The approach is novel and potentially interesting, 
      but is fairly complicated and not very clearly described in the paper. 
      While the analogy between image patches and word groups of 
      co-occurring words is probably worth mentioning, using the image patch 
      terminology throughout the paper is more confusing than helpful. 
      The experimental section is the weakest point of the paper. The 
      authors claim that their model performs so well because it is 
      hierarchical and nonlinear, but the experimental results do not 
      provide direct evidence for these claims. To substantiate these 
      claims, DeepMatch should be compared to a linear version of itself 
      (with all sigmoid units replaced by linear ones) as well as to a 
      non-hierarchical version of itself (using only one resolution of 
      topics). Without these comparisons there are several confounding 
      factors that could potentially explain the superior results obtained 
      by DeepMatch. As for using existing baselines, it makes more sense to 
      compare to Supervised Semantic Indexing or Polynomial Semantic 
      Indexing instead of RMLS, as those methods are trained using the same 
      margin-based criterion as DeepMatch. The choice of the datasets is 
      also unfortunate, as they appear to be non-standard and proprietary, 
      which makes the reported scores difficult to interpret and the results 
      impossible to reproduce. The authors also do not report several 
      crucial details such as the latent dimensionality of the models, the 
      number of topics used, the number hidden units (sigmoid vs. linear), 
      regularization parameters etc. The description of the hyperparameter 
      selection procedure on lines 345-48 is a cause for concern as it seems 
      to suggest that the hyperparemeters where selected on a subset of the 
      training set, which is likely to result in overfitting. The big gap in 
      performance between DeepMatch and the siamese network makes me wonder 
      whether the latter was properly regularized. The mention of "zero 
      regularization" for that model on line 364 seems to suggest otherwise. 
      Given the relatively small size of the datasets for the vocabulary 
      sizes used, proper model regularization is absolutely essential for a 
      fair comparison. The gradients given by Eqs. (8) and (9) are not 
      quite right as they reference y_i instead of y+_i and y-_i. It is 
      unclear what the potential value (pot^k_p) is. Is it the total input 
      to a unit? It appears that DeepMatch used a reduced vocabulary 
      ("trim the words" on line 172). Is this correct and if so, was the 
      same vocabulary reduction performed for all models? How 
      exactly are the higher-resolution topics are assigned to the 
      lower-resolution ones? Were the comparison accuracy scores on 
      Figure 6 computed on the training data or on the test data? What are 
      the "correct comparisons" shown on the figure? The paper 
      repeatedly uses "localness" instead of the more appropriate 
      "locality".