A Novel Two-Step Method for Cross Language Representation Learning
Cross language text classification is an important learning task in natural language processing. A critical challenge of cross language learning arises from the fact that words of different languages are in disjoint feature spaces. In this paper, we propose a two-step representation learning method to bridge the feature spaces of different languages by exploiting a set of parallel bilingual documents. Specifically, we first formulate a matrix completion problem to produce a complete parallel document-term matrix for all documents in two languages, and then induce a low dimensional cross-lingual document representation by applying latent semantic indexing on the obtained matrix. We use a projected gradient descent algorithm to solve the formulated matrix completion problem with convergence guarantees. The proposed method is evaluated by conducting a set of experiments with cross language sentiment classification tasks on Amazon product reviews. The experimental results demonstrate that the proposed learning method outperforms a number of other cross language representation learning methods, especially when the number of parallel bilingual documents is small.
The paper proposes a matrix completion approach to the 
      cross domain classification task, which is capable of exploiting labeled 
      data in an auxiliary domain and also unlabeled parellel data. The approach 
      involves two steps. First, it constructs a single incomplete matrix that 
      includes all documents and domains, and then completes this matrix while 
      enforcing low-rank and sparsity conditions using the projected gradient 
      descent algorithm. Second, it reduces the feature dimension of this 
      completed matrix using LSI, and then trains a standard classifier on this 
      new representation. The paper also presents a convergence guarantee on the 
      projected gradient descent algorithm, and favorable experimental results. 
      Here are some comments and questions: - After estimating 
      the completed matrix M*, you reduce the feature dimension from d to k with 
      LSI because M* "lacks sufficient capacity of handling feature sparseness". 
      Can you elaborate on this? Is it basically that you want small dimensions 
      for computational convenience, since M* will be relatively dense even with 
      the sparsity enforcement? Also, in that case, is there a reason you chose 
      LSI over other dimensionality reduction techniques (e.g., PCA)? - 
      There seem to be two disjoint aspects of exploiting external data in this 
      paper: 1. leveraging the labeled source language data to help learn the 
      labeled target language data, and 2. leveraging the unlabeled parallel 
      data. If this is correct, it would be helpful to clearly distinguish the 
      two and report results separately. - On a related note, the 
      experiment settings are a bit unclear (section 5.2). So you were using (a) 
      4k labeled source language documents, (b) 100 labeled target language 
      documents, and (c) 2k unlabeled parallel documents for your training data? 
      And you are reporting the classification results on the remaining 1900 
      labeled target language documents? Then the only difference in setion 5.3 
      is that you are varying the number of unlabeled parallel documents from 
      200 to 2k? In that case, why is the performance of CL-KCCA better than TSL 
      in Figure 1 for EFM and EGM at 2k but it is not in Table 1? - Can 
      you briefly clarify why a fully observed document-term matrix would be 
      low-rank? It is sparse, but probably no document is a linear combination 
      of other documents in vocabulary, meaning the rank in that case will be 
      min(d, k). - Can you give an intuition on why TSL is able to 
      perform better in general than other methods, especially CCA? A multiview 
      approach like CCA feels like a natural approach to this cross language 
      representation learning task. In contrast, the matrix completion approach 
      is not as natural. It seems the main driving force is enforcing the 
      low-rank condition on M*. How does this compare with other techniques? 
      - What is the running time of the algorithm? Does the projected 
      gradient descent algorithm take more, less, or about the same time as 
      other methods like CL-LSI, CL-KCCA, and CL-OPCA? Here are some 
      suggested changes: - Fix (ij) to (i,j) two sentences before 
      equation (2). - Remove "such as" in the sentences before equation (4) 
      and (5).