We introduce a theoretical analysis of crosslingual transfer in probabilistic topic models. By formulating posterior inference through Gibbs sampling as a process of language transfer, we propose a new measure that quantifies the loss of knowledge across languages during this process. This measure enables us to derive a PAC-Bayesian bound that elucidates the factors affecting model quality, both during training and in downstream applications. We provide experimental validation of the analysis on a diverse set of five languages, and discuss best practices for data collection and model design based on our analysis. Crosslingual learning is an important area of natural language processing that has driven applications including text mining in multiple languages ( Ni et al., 2009; Smet and Moens, 2009), cultural difference detection (Guti  errez et al., 2016), and various linguistic studies (Shutova et al. , 2017; Barrett et al., 2016). Crosslingual learning methods generally extend monolingual algorithms by using various multilingual resources. In contrast to traditional high-dimensional vector space models, modern crosslingual models tend to rely on learning low-dimensional word representations that are more efficient and generalizable. A popular approach to representation learning comes from the word embedding community, in which words are represented as vectors in an embedding space shared by multiple languages (Ruder et al., 2018; Faruqui and Dyer, 2014; Klementiev et al., 2012). Another direction is from the topic modeling community, where words are projected into a probabilistic topic space (Ma and Nasukawa, 2017; Jagarlamudi and III, 2010). While formulated differently, both types of models apply the same principles low-dimensional vectors exist in a shared crosslingual space, wherein vector representations of similar concepts across languages ( e.g., dog and hund) should be nearby in the shared space. To enable crosslingual representation learning, knowledge is transferred from a source language to a target language, so that representations have similar values across languages. In this study, we will focus on probabilistic topic models, and knowledge refers to a word's probability distribution over topics. Little is known about the characteristics of crosslingual knowledge transfer in topic models, and thus this paper provides an analysis, both theoretical and empirical, of crosslingual transfer in multilingual topic models. Multilingual Topic Models Given a multilingual corpus D (1 ;:::;L ) in languages  = 1 ; : : : ; L as inputs, a multilingual topic model learns K topics. Each multilingual topic k (1 ;:::;L ) ( k = 1 ; : : : ; K ), is defined as an L -dimensional tuple (  (1) k ; : : : ;  ( L ) k ) , where  (  ) k is a multinomial distribution over the vocabulary V (  ) in language  . From a human's perspective, a multilingual topic k (1 ;:::;L ) can be interpreted by looking at the word types that have C highest probabilities in  (  ) k for each language  . C here is called cardinality of the topic. Thus, a multilingual topic can loosely be thought of as a group of word lists where each language  has its own version of the topic. Multilingual topic models are generally extended from Latent Dirichlet Allocation (Blei et al., 2003, LDA ). Though many variations have been proposed, the underlying structures of multilingual topic models are similar. These models require either a parallel/comparable corpus in multiple languages, or word translations from a dictionary. One of the most popular models is the polylingual topic model (Mimno et al., 2009, PLTM ), where comparable document pairs share distributions over topics (cid:18) , while each language  has its own distributions f  (  ) k g Kk =1 over the vocabulary V (  ) . By re-marginalizing the estimations f b  (  ) k g Kk =1 , we obtain word representations b  ( w ) 2 RK for each word w , where b  ( w ) k = Pr( z w = k j w ) , i.e., the probability of topic k given a word type w . Crosslingual Transfer Knowledge transfer through crosslingual representations has been studied in prior work. Smet and Moens (2009) and Heyman et al. (2016) show empirically how document classification using topic models implements the ideas of crosslingual transfer, but to date there has been no theoretical framework to analyze this transfer process in detail. In this paper, we describe two types of transferon-site and off-sitebased on the na-ture of where and how the transfer takes place. We refer to transfer that happens while training topic models ( i.e., during representation learning) as on-site . Once we obtain the low-dimensional representations, they can be used for downstream tasks. We refer to transfer in this phase as off-site , since the crosslingual tasks are usually detached from the process of representation learning. Contributions Our study provides a theoretical analysis of crosslingual transfer learning in topic models. Specifically, we first formulate on-site transfer as circular validation, and derive an upper bound based on PAC-Bayesian theories (Sec-tion 2). The upper bound explicitly shows the factors that can affect knowledge transfer. We then move on to off-site transfer, and focus on crosslingual document classification as a downstream task (Section 3). Finally, we show experimentally that the on-site transfer error can have impact on the performance of downstream tasks (Section 4). On-site transfer refers to the training procedure of multilingual topic models, which usually involves Bayesian inference techniques such as variational inference and Gibbs sampling. Our work focuses on the analysis of collapsed Gibbs sampling (Grif-fiths and Steyvers, 2004), showing how knowledge is transferred across languages and how a topic space is formed through the sampling process. To this end, we first describe a specific formulation of knowledge transfer in multilingual topic models as a starting point of our analysis (Sec-tion 2.1). We then formulate Gibbs sampling as circular validation and quantify a loss during this phase (Section 2.2). This formulation leads us to a PAC-Bayesian bound that explicitly shows the factors that affect the crosslingual training (Sec-tion 2.3). Lastly, we look further into different transfer mechanisms in more depth (Section 2.4). Priors are an important component in Bayesian models like PLTM . In the original generative process of PLTM , each comparable document pair ( d S ; d T ) in the source and target languages ( S; T ) is generated by the same multinomial (cid:18) (cid:24) Dir( (cid:11) ) . Hao and Paul (2018) showed that knowledge transfer across languages happens through priors. Specifically, assume the source document is generated from (cid:18) ( d S ) (cid:24) Dir( (cid:11) ) , and has a sufficient statistics n d S 2 NK where each cell n k j d S is the count of topic k in document d S . When generating the corresponding comparable document d T , the Dirichlet prior of the distribution over topics (cid:18) ( d T ) , instead of a symmetric (cid:11) , is parameterized by (cid:11) + n d S . This formulation yields the same posterior estimation as the original joint model and is the foundation of our analysis in this section. To see this transfer process more clearly, we look closer to the conditional distributions during sampling, and take PLTM as an example. When sampling a token in target language x T , the Gibbs sampler calculates a conditional distribution P x T over K topics, where a topic k is randomly drawn and assigned to x T (denoted as z x T ). Assume the token x T is in document d T whose comparable document in the source language is d S . The conditional distribution for x T is P x;k = Pr( z x = k ; w (cid:0) ; z (cid:0) ) (1) / ( n k j d T + n k j d S + (cid:11) ) (cid:1) n w T j k + (cid:12) n (cid:1)j k + V ( T ) (cid:12) ; where the quantity n k j d S is added and thus transferred from the source document. Thus, the calculation of P x incorporates the knowledge transferred from the other language. Now that we have identified the transfer process, we provide an alternative view of Gibbs sampling, i.e., circular validation, in the next section. Circular validation (or reverse validation) was proposed by Zhong et al. (2010) and Bruzzone and Marconcini (2010) in transfer learning. Briefly, a learning algorithm A is trained on both source and target datasets ( DS and DT ), where the source is labeled and target is unlabeled. After predicting the labels for the target dataset using A (predic-tions denoted as A ( DT ) ), circular validation trains another algorithm A  in the reverse direction, i.e., uses A ( DT ) and DT as the labeled dataset and DS as the unlabeled dataset. The error is then evaluated on A  ( DS ) . This train-predict-reverse-repeat cycle has a similar flavor to the iterative manner of Gibbs sampling, which inspires us to look at the sampling process as circular validation. Figure 1 illustrates this process. Suppose the Gibbs sampler is currently sampling x T of word type w T in target language T . As discussed for Equation (1), the calculation of the conditional distribution P x T incorporates the knowledge transferred from the source language. We then treat the process of drawing a topic from P x T as a classification of the token x T . Let P x T be a distribution over K unary classifiers , f h k g Kk =1 , and the k -th classifier labels the token as topic k with a probability of one: h k (cid:24) P x T ; and Pr ( z x T = k ; h k ) = 1 : (2) This process is repeated between the two languages until the Markov chain converges. The training of topic models is unsupervised, i.e., there is no ground truth for labeling a topic, which makes it difficult to analyze the effect of transfer learning. Thus, after calculating P x T , we take an additional step called reverse validation , where we design and calculate a measure circular validation lossto quantify the transfer. Definition 1 (Circular validation loss, CVL ) . Let S w be the set containing all the tokens of type w throughout the whole training corpus, and call it the sample of w . Given a bilingual word pair ( w T ; w S ) where w T is in target language T while w S in source S , let S w T and S w S be the samples for the two types respectively, and n w T and n w S the sizes of them. The empirical circular validation score ( d CVL ) is defined as d CVL ( w T ; w S ) = 1 2 E x S ;x T [ b L ( x T ; w S ) + b L ( x S ; w T ) ] ; b L ( x T ; w S ) = 1 n w S  x S 2S wS E h (cid:24)P xT [ 1 f h ( x S )  = z x S g ] = 1 n w S  x S 2S wS ( 1 (cid:0) P x T ;z xS ) ; where P x T ;k is the conditional probability of token x T assigned with topic k . When sampling a token x T , we still follow the two-step process as in Equation (2), but instead of labeling x T itself, we use its conditional P x T to label the entire sample of a word type w S in the source language. Since all the topic labels for the source language are fixed, we take them as the assumed correct labelings, and compare x S 's labels and the predictions from P x T . This is the intuition behind CVL . Note that the choice of word types w T and w S to calculate d CVL is arbitrary. However, d CVL is only meaningful when the two word types are semantically related, such as word translations, because those word pairs are where the knowledge transfer takes place. On the other hand, the Gibbs sampler does not calculate this d CVL explicitly, and thus adding reverse validation step does not affect the training of the model. It does, however, help us to expose and analyze the knowledge transfer mechanism. In fact, as we show in the next theorem, sampling is also a procedure of optimizing d CVL . Theorem 1. Let d CVL ( t ) ( w T ; w S ) be the empirical circular validation loss of any bilingual word pair at iteration t of Gibbs sampling. Then d CVL ( t ) ( w T ; w S ) converges as t ! 1 . A question following the formulation of d CVL is, what factors could lead to better transfer during this process, particularly for semantically related words? To answer this, we turn to theory that bounds the performance of classifiers and apply this theory to this formulation of topic sampling as classification. The PAC-Bayes theorem was introduced by McAllester (1999) to bound the performance of Bayes classifiers. Given a hypothesis set H , the majority vote classifier (or Bayes classifier) uses every hypothesis h 2 H to perform binary classification on an example x , and uses the majority output as the final prediction. Since minimizing the error by Bayes classifier is NP-hard, an alternative way is to use a Gibbs classifier as approximation. The Gibbs classifier first draws a hypothesis h 2 H according to a posterior distribution over H , and then uses this hypothesis to predict the label of an example x ( Germain et al., 2012). The generalization loss of this Gibbs classifier can be bounded as follows. Theorem 2 (PAC-Bayes theorem, McAllester (1999)) . Let P be a posterior distribution over all classifiers h 2 H , and Q a prior distribution. With a probability at least 1 (cid:0) (cid:14) , we have L (cid:20) b L +  1 2 n ( KL ( Pjj Q ) + ln 2 p n (cid:14) ) ; where L and b L are the general loss and the empirical loss on a sample of size n . In our framework, a token x T provides a posterior P x T over K classifiers. The loss b L ( x T ; w S ) is then calculated on a sample of S w S in language S . The following theorem shows that for a bilingual word pair ( w T ; w S ) , the general CVL can be bounded with several quantities. For brevity we use KL w to denote KL( P x jj Q x ) , where P x is the conditional distribution from Gibbs sampling of token x with word type w that gives highest loss b L ( x; w ) , and Q x a prior. Proof. See Appendix. Recall that knowledge transfer happens through priors in topic models (Section 2.1). Because the KL-divergence terms in Theorem 3 include this prior Q , we can use this theorem to analyze the transfer mechanisms more concretely. The conditional distribution for sampling a topic z x for a token x during sampling can be factorized into document-topic and topic-word levels: P x;k = Pr ( z x = k j w x = w; w (cid:0) ; z (cid:0) ) = Pr ( z x = k j z (cid:0) ) (cid:1) Pr ( w x = w j z x = k; w (cid:0) ; z (cid:0) ) / Pr ( z x = k j z (cid:0) ) | {z } documentlevel (cid:1) Pr ( z x = k j w x = w; w (cid:0) ) | {z } wordlevel  = P (cid:18);x;k (cid:1) P ;x;k ; P x  = P (cid:18);x (cid:10) P ;x ; where (cid:10) is element-wise multiplication. Thus, we have the following inequality: KL ( P x jj Q x ) = KL ( P (cid:18);x (cid:10) P ;x jj Q (cid:18);x (cid:10) Q ;x ) (cid:20) KL ( P (cid:18);x jj Q (cid:18);x ) + KL ( P ;x jj Q ;x ) ; and the KL-divergence term in Theorem 3 is simply the sum of the KL-divergences between the conditional and prior distributions on all levels. Recall that PLTM transfers knowledge at the document level, through Q (cid:18);x , by linking document translations together (Equation (1)). Assume the current token x is from a target document linked to a document d S in the source language. Then the prior for P (cid:18);x is b (cid:18) ( d S ) , i.e., the normalized empirical distribution over topics of d S . Since the words are generated within each language under PLTM , i.e.,  ( S ) k is irrelevant to  ( T ) k , no transfer happens at the word level. In this case, Q ;x , the prior for P ;x , is simply a K dimensional uniform distribution U . Then: KL w (cid:20) KL ( P (cid:18);x jj b (cid:18) ( d S ) ) + KL ( P ;x jjU ) = KL ( P (cid:18);x jj b (cid:18) ( d S ) ) | {z } crosslingual entropy + log K (cid:0) H ( P ;x ) | {z } monolingual entropy : Thus, at levels where transfer happens (document-or word-level), a low crosslingual entropy is preferred, to offset the impact of monolingual entropy where no transfer happens. Most multilingual topic models are generative admixture models in which the conditional probabilities can be factorized into different levels, thus KL-divergence term in Theorem 3 can be decomposed and analyzed in the same way as in this section for models that have transfer at other levels, such as Hao and Paul (2018), Heyman et al. (2016), and Hu et al. (2014). For example, if a model has word-level transfer, i.e., the model assumes that word translations share the same distributions, we have a KL-divergence term as, KL w (cid:20) KL ( P ;x jj b  ( w S ) ) + KL( P (cid:18);x jjU ) = KL ( P ;x jj b  ( w S ) ) + log K (cid:0) H ( P (cid:18);x ) ; where w S is the word translation to word w . Off-site transfer refers to language transfer that happens while applying trained topic models to downstream crosslingual tasks such as document classification. Because transfer happens using the trained representations, the performance of off-site transfer heavily depends on that of on-site transfer. To analyze this problem, we focus on the task of crosslingual document classification. In crosslingual document classification, a document classifier , h , is trained on documents from one language, and h is then applied to documents from another language. Specifically, after training bilingual topic models, we have K bilingual word distributions f b  ( S ) k g Kk =1 and f b  ( T ) k g Kk =1 . These two distributions are used to infer document-topic distributions b (cid:18) on unseen documents in the test corpus, and each document is represented by the inferred distributions. A document classifier is then trained on the b (cid:18) vectors as features in source language S and tested on the target T . We aim to show how the generalization risk on target languages T , denoted as RT ( h ) , is related to the training risk on source languages S , c RS ( h ) . To differentiate the loss and classifiers in this section from those in Section 2, we use the term risk here, and h refers to the document classifiers, not the topic labeling process by the sampler. Classic learning theory requires training and test sets to come from the same distribution D , i.e., ( (cid:18); y ) (cid:24) D , where (cid:18) is the document representation (features) and y the document label (Valiant, 1984). In practice, however, corpora in S and T may be sampled from different distributions, i.e., D ( S ) = f ( b (cid:18) ( d S ) ; y ) g (cid:24) b D ( S ) and D ( T ) = f ( b (cid:18) ( d T ) ; y ) g (cid:24) b D ( T ) . We refer to these distributions as document spaces . To relate RT ( h ) and c RS ( h ) , therefore, we have to take their distribution bias into consideration. This is often formulated as a problem of domain adaptation, and here we can formulate this such that each language is treated as a domain. We follow the seminal work by Ben-David et al. (2006), and define H -distance as follows. Definition 2 ( H -distance, Ben-David et al. (2006)) . Let H be a symmetric hypothesis space, i.e., for every hypothesis h 2 H there exists its counterpart 1 (cid:0) h 2 H . We let m = (cid:12)(cid:12) D ( S ) (cid:12)(cid:12) + (cid:12)(cid:12) D ( T ) (cid:12)(cid:12) , the total size of test corpus. The H distance between b D ( S ) and b D ( T ) is defined as 1 2 b d H ( b D ( S ) ; b D ( T ) ) = max h 2H 1 m   2f S;T g  x d : h ( x d )=  1 { x d 2 D (  ) } ; where x d is the representation for document d , and h ( x d ) outputs the language of this document. This distance measures how identifiable the languages are based on their representations. If source and target languages are from entirely different distributions, a classifier can easily identify language-specific features, which could affect performance of the document classifiers. With H -distances, we have a measure of the distance between the two distributions b D ( S ) and b D ( T ) . We state the following theorem from domain adaptation theory. Theorem 4 (Ben-David et al. (2006)) . Let m be the corpus size of the source language, i.e., m = (cid:12)(cid:12) D ( S ) (cid:12)(cid:12) , c the VC-dimension of document classifiers h 2 H , and b d H ( b D ( S ) ; b D ( T ) ) the H -distance between two languages in the document space. With probability at least 1 (cid:0) (cid:14) , we have the following bound, RT ( h ) (cid:20) b RS ( h ) + b d H ( b D ( S ) ; b D ( T ) ) + b (cid:21) +  4 m ( c log 2 em c + log 4 (cid:14) ) ; (4) b (cid:21) = min h 2H b RS ( h ) + b RT ( h ) : (5) The term b (cid:21) in Theorem 4 defines a joint risk , i.e., the training error on both source and target documents. This term usually cannot be estimated in practice since the labels for target documents are unavailable. However, we can still calculate this term for the purpose of analysis. The theorem shows that the crosslingual classification risk is bounded by two critical components: the H -distance, and the joint risk b (cid:21) . Interestingly, these two quantities are based on the same set of features with different labeling rules: for H -distance, the label for each instance is its language, while b (cid:21) uses the actual document label. Therefore, a better bound requires the consistency of features across languages, both in language and document labelings. Since consistency of features depends on the document representations b (cid:18) , we need to trace back to the upstream training of topic models and show how the errors propagate to the formation of document representations. Thus, we first show the relations between d CVL and word representations b  in the following lemma. Lemma 1. Given any bilingual word pair ( w T ; w S ) , let b  ( w ) denote the distribution over topics of word type w . Then we have, 1 (cid:0) b  ( w T )  (cid:1) b  ( w S ) (cid:20) d CVL ( w T ; w S ) : Proof. We need to connect the word representations b  , which are central to on-site transfer, to the document representations b (cid:18) , which are central to off-site transfer. To do this, we make an assumption that the inferred distribution over topics b (cid:18) ( d ) for each test document d is a weighted average over all word vectors, i.e., b (cid:18) ( d ) /  w f dw (cid:1) b  ( w ) , where f dw is the normalized frequency of word w in document d (Arora et al., 2013). When this assumption holds, we can bound the similarity of document representations b (cid:18) ( d S ) and b (cid:18) ( d T ) in terms of word representations and hence their d CVL . Theorem 5. Let b (cid:18) ( d S ) be the distribution over topics for document d S (similarly for d T ), F ( d S ; d T ) = ( w S f d S w S 2 (cid:1)  w T f d T w T 2 ) 12 where f dw is the normalized frequency of word w in document d , and K the number of topics. Then b (cid:18) ( d S )  (cid:1) b (cid:18) ( d T ) (cid:20) F ( d S ; d T ) (cid:1)  K (cid:1)  w S ;w T ( d CVL ( w T ; w S ) (cid:0) 1 ) 2 : Proof. This provides a spatial connection between document pairs and word pairs they have. Many ker-nalized classifiers such as support vector machines ( SVM ) explicitly use this inner product in the dual optimization objective (Platt, 1998). Since the inner product is directly related to the cosine similarity, Theorem 5 indicates that if two documents are spatially close, their inner product should be large, and thus the d CVL of all word pairs they share should be small. In an extreme case, if d CVL ( w T ; w S ) = 1 for all the bilingual word pairs appearing in document pair ( d S ; d T ) , then b (cid:18) ( d S )  (cid:1) b (cid:18) ( d T ) = 0 , meaning the two documents are orthogonal and tend to be irrelevant topically. With upstream training discussed in Section 2, we see that d CVL has an impact on the consistency of features across languages. A low d CVL indicates that the transfer from source to target is sufficient in two ways. First, languages share similar distributions, and therefore, it is harder to distinguish languages based on their distributions. Second, if there exists a latent mapping from a distribution to a label, it should produce similar labeling on both source and target data since they are similar. These two aspects correspond to the language H distance and joint risk b (cid:21) in Theorem 4. 4 Experiments We experiment with five languages: Arabic ( AR , Semitic), German ( DE , Germanic), Spanish ( ES , Romance), Russian ( RU , Slavic), and Chinese ( ZH , Sinitic). In the first two experiments, we pair each with English ( EN , Germanic) and train PLTM on each language pair individually. Training Data For each language pair, we use a subsample of 3 ; 000 Wikipedia comparable documents, i.e., 6 ; 000 documents in total. We set K = 50 , and train PLTM with default hyperparameters (McCallum, 2002). We run each experiment five times and average the results. Test Data For experiments with document classification, we use Global Voices ( GV ) in all five language pairs as test sets. Each document in this dataset has a categories attribute that can be used as the document label. In our classification experiments, we use culture , technology , and education as the labels to perform multiclass classification. Evaluation To evaluate topic qualities, we use Crosslingual Normalized Pointwise Mutual Information (Hao et al., 2018, CNPMI ), an intrinsic metric of crosslingual topic coherence. For any bilingual word pair ( w T ; w S ) , CNPMI ( w T ; w S ) = (cid:0) log Pr( w T ;w S ) Pr( w T )Pr( w S ) log Pr ( w T ; w S ) ; (6) where Pr ( w T ; w S ) is the occurrence of w T and w S appearing in the same pair of comparable documents. We use 10 ; 000 Wikipedia comparable document pairs outside PLTM training data for each language pair to calculate CNPMI scores. All datasets are publicly available at http:// opus.nlpl.eu/ (Tiedemann, 2012). Additional details of our datasets and experiment setup can be found in the appendix. Our first experiment shows how d CVL changes over time during Gibbs sampling. According to the definition, the arguments of d CVL can include any bilingual word pairs; however, we suggest that it should be calculated specifically among word pairs that are expected to be related (and thus enable transfer). In our experiments, we select word pairs in the following way. Recall that the output of a bilingual topic model is K topics, where each language has its own distribution. For each topic k , we can calculate d CVL ( w S ; w T ) such that w S and w T belong to the same topic ( i.e., are in the top C most probable words in that topic), from the two languages, respectively. Using a cardinality C for each of the K topics, we have in total C 2 (cid:2) K bilingual word pairs in the calculation of d CVL . At certain iterations, we collect the topic words as described above with cardinality C = 5 , and calculate d CVL ( w T ; w S ) , CNPMI ( w T ; w S ) , and the error term (the 12 p(cid:1) (cid:1) (cid:1) term in Theorem 3) of all the bilingual word pairs. In the middle panel of Figure 2, d CVL over all word pairs from topic words is decreasing as sampling proceeds and becomes stable by the end of sampling. On the other hand, the correlations between CNPMI and d CVL are constantly decreasing. The negative correlations between d CVL and CNPMI implies that lower d CVL is associated with higher topic quality, since higher-quality topic has higher CNPMI but lower d CVL . Theorem 3 provides insights into how knowledge is transferred during sampling and the factors that could affect this process. We analyze this bound from two aspects, the size of the training data (cor-responding to ln n  n term) and model assumptions (as in the crosslingual entropy terms). One factor that could affect d CVL , according to Theorem 3, is the balance of tokens of a word pair. In an extreme case, if a word type w S has only one token, while another word type w T has a large number of tokens, the transfer from w S to w T is negligible. In this experiment, we will test if increasing the ratio term ln n  n in the corpus lowers the performance of crosslingual transfer learning. To this end, we specify a sample rate (cid:26) = 0 : 2 ; 0 : 4 ; 0 : 6 ; 0 : 8 ; and 1 : 0 . For each word pair ( w T ; w S ) , we calculate n as in the ratio term ln n  n , and remove (1 (cid:0) (cid:26) ) (cid:1) n tokens from the corpus (rounded to the nearest integer). Smaller (cid:26) removes more tokens from the corpus and thus yields a larger ratio term on average. We use a dictionary from Wiktionary to collect word pairs, where each word pair ( w S ; w T ) is a translation pair. Figure 3 shows the results of downsampling using these two methods. Decreasing the sample rate (cid:26) lowers the topic qualities. This implies that although PLTM can process comparable corpora, which need not be exact translations, one still needs to be careful about the token balance between linked document pairs. For many low-resource languages, the target language corpus is much smaller than the source corpus, so the effect of this imbalance is important to be aware of. This is an important issue when choosing comparable documents, and Wikipedia is an illustrative example. Although one can collect comparable documents via Wikipedia's interlanguage links, articles under the same title but in different languages can have very large variations on document length, causing the imbalance of samples ln n  n , and thus potentially suboptimal performance of crosslingual training. Recall that the crosslingual entropy term can be decomposed into different levels, e.g., document level and word level, and we prefer a model with low crosslingual entropy but high monolingual entropy. In this experiment, we show how these two quantities affect the topic qualities, using English-German ( EN-DE ) documents as an example. Given PLTM output in ( EN , DE ) and a cardinality C = 5 , we collect C 2 (cid:2) K bilingual word pairs as described in Section 4.1. For each word pair, we calculate three quantities: d CVL , CNPMI , and the inner product of the word representations. In Figure 4, each dot is a word pair ( w S ; w T ) colored by the values of these quantities. The word pair dots are positioned by their crosslingual and monolingual entropies. We observe that d CVL decreases with crosslingual entropy on document level. The larger the crosslingual entropy, the harder it is to get a low d CVL because it needs larger monolingual entropy to decrease the bound, as shown in Section 2.4. On the other hand, the inner product of word pairs shows an opposite pattern of d CVL , indicating a negative correlation (Lemma 1). see the correlation between CNPMI and d CVL is around (cid:0) 0 : 4 at the end of sampling, so there are fewer clear patterns for CNPMI in Figure 4. However, we also notice that the word pairs with higher CNPMI scores often appear at the bottom where crosslingual entropy is low while the monolingual entropy is high. We move on to crosslingual document classification as a downstream task. At various iterations of Gibbs sampling, we infer topics on the test sets for another 500 iterations and calculate the quantities shown in the Figure 5 (averaged over all lan-guages), including the H -distances for both training and test sets, and the joint risk b (cid:21) . We treat English as the source language and train support vector machines to obtain the best classifier h  that fits the English documents. This classifier is then used to calculate the source and target risks b RS ( h  ) and b RT ( h  ) . We also include 12 b d H ( S; T ) , the H -distance based on word rep-1 10 20 40 60 80 100 500 1000 Iterations 0 . resentations b  . As mentioned in Section 3.1, we train support vector machines to use languages as labels, and the accuracy score as the H -distance. The classification risks, such as b RS ( h  ) , b RT ( h  ) , and b (cid:21) , are decreasing as expected (upper row in Figure 5), which shows very similar trends as d CVL in Figure 2. On the other hand, we notice that the H -distances of training documents and vocabularies, 12 b d H ( b D ( S ) ; b D ( T ) ) and 12 b d H ( S; T ) , stabilize around 0 : 5 to 0 : 6 , meaning it is difficult to differentiate the languages based on their representations. Interestingly, the H -distances of test documents are at a less ideal value, although they are slightly decreasing in most of the languages except AR . However, recall that the target risk also depends on other factors than H -distance (Theo-rem 4), and we use Figure 6 to illustrate this point. We further explore the relationship between the predictability of languages vs document classes in Figure 6. We collect documents correctly classi-fied for both document class and language labels, from which we randomly choose 200 documents for each language, and use b (cid:18) to plot t-SNE scatterplots. Note that the two plots are from the same set of documents, and so the spatial relations between any two points are fixed, but we color them with different labelings. Although the classifier can identify the languages (right panel), the features are still consistent, because on the left panel, the decision boundary changes its direction and also successfully classifies the documents based on actual label class. This illustrates why a single H -distance does not necessarily mean inconsistent features across languages and high target risks. This study gives new insights into crosslingual transfer learning in multilingual topic models. By formulating the inference process as a circular validation, we derive a PAC-Bayesian theorem to show the factors that affect the success of crosslingual learning. We also connect topic model learning with downstream crosslingual tasks to show how errors propagate. As the first step toward more theoretically justi-fied crosslingual transfer learning, our study suggests considerations for constructing crosslingual transfer models in general. For example, an effective model should strengthen crosslingual transfer while minimizing non-transferred components, use a balanced dataset or specific optimization algorithms for low-resource languages, and support evaluation metrics that relate to CVL .
abstain objective abstain method abstain abstain abstain abstain abstain abstain abstain method method other other other other other other other other other result other other method method result abstain objective objective other method result abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain other method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method other other abstain abstain abstain abstain abstain other abstain method abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain method method method abstain abstain abstain other abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain method abstain method method other method other abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain objective result result objective abstain