Recent years have witnessed growing interests in incorporating external knowledge such as pre-trained word embeddings (PWEs) or pre-trained language models (PLMs) into neural topic modeling. However, we found that employing PWEs and PLMs for topic modeling only achieved limited performance improvements but with huge computational overhead. In this paper, we propose a novel strategy to incorporate external knowledge into neural topic modeling where the neural topic model is pre-trained on a large corpus and then fine-tuned on the target dataset. Experiments have been conducted on three datasets and results show that the proposed approach significantly outperforms both current state-of-the-art neural topic models and some topic modeling approaches enhanced with PWEs or PLMs. Moreover, further study shows that the proposed approach greatly reduces the need for the huge size of training data. Topic models have been widely used for discovering hidden themes from a large collection of documents in an unsupervised manner. Recently, to avoid the complex and specific inference process of graph model-based method such as LDA (Blei et al., 2003), neural topic modeling that utilizes neural-network-based black-box inference has been the main research direction in this field (Blei, 2012; Miao et al., 2016; Srivastava and Sutton, 2017). Typically, neural topic models infer topics of a document by utilizing its bag-of-words (BoWs) representation to capture word co-occurrence patterns. The BoWs representation, however, fails to encode rich word semantics, leading to relatively inferior quality of topics generated by the topic models. Therefore, approaches have been proposed to address the limitation of BoWs representation  Corresponding author. by incorporating the external knowledge, such as pre-trained word embeddings (PWEs) (Das et al., 2015; Wang et al., 2020; Dieng et al., 2020). In recent years, pre-trained language models (PLMs) (Peters et al., 2018; Devlin et al., 2019; Brown et al., 2020) have achieved state-of-the-art performance on a wide range of natural language processing tasks. Different from PWEs 1 in which a word is mapped to a static word emebdding, PLMs generate a specific word embedding for each occurrence of a word depending on the context. It is appealing to incorporate PLMs into topic models since contextualized embeddings generated by PLMs encode richer semantics and naturally deal with word polysemy (Pasini et al., 2020). One straightforward way is to replace BoWs representation with the outputs of PLM (Bianchi et al., 2020b) in existing topic models or take PLM outputs as additional inputs to topic modeling (Bianchi et al., 2020a). A more sophisticated approach is to distill the knowledge of a PLM into a topic model. For example, (Hoyle et al., 2020) employed the probability estimates of a teacher PLM over a text sequence to guide the training of a student topic model. However, the approaches mentioned above still have limitations. Firstly, using PLMs for topic model training in such ways leads to huge computational overhead. Most neural topic models are based on shallow multi-layer perceptions with few hidden units. However, most popular PLMs are based on deep Transformers (Vaswani et al., 2017) where at each layer expensive self-attention operations are performed, which have a time complexity quadratic in document length. Therefore, the overall training time is dominated by PLM, and it will be worse if PLM is further fine-tuned, as shown in (Hoyle et al., 2020). Secondly, there is the gap of training objectives between PLMs and topic models, where PLMs are trained to learn the semantic 1 In this paper, PWEs refer to context-free embeddings. and syntactic knowledge within a sentence while topic models focus on extracting main themes over whole corpus. As shown in Table 4, a model based on GloVe embeddings (Pennington et al., 2014) performs better than PLMs-based models such as those proposed in (Bianchi et al., 2020a) and (Bianchi et al., 2020b). To overcome these challenges, we propose a simple yet effective strategy, namely Pre-trained Neural Topic Model (PT-NTM), to utilize extensive knowledge from large corpora for neural topic modeling with low computational complexity. Instead of pre-training the embeddings and acquiring knowledge indirectly, PT-NTM directly pre-trains the topic model itself on the knowledge source corpora. In specific, a neural topic model is firstly trained on a large corpus only once, which is called pre-training . Afterward, it is fine-tuned on any other dataset, which is called fine-tuning . As the architecture of the neural topic model used in pretraining and fine-tuning is the same, it incurs little computational overhead to any subsequent training. Experiments have been conducted on three datasets and the results show that the proposed approach significantly outperforms not only some state-of-the-art neural topic models but also the topic modeling approaches using PWEs and PLMs. Moreover, it is observed that on the NYTimes dataset, the neural topic model trained on 1% of the whole dataset using the proposed approach achieves superior performance than other baseline models that are trained on the whole dataset. It further shows that the proposed approach greatly reduces the need for the huge size of training data. The main contributions are:  We proposed a simple yet effective strategy for training neural topic models in which the models are pre-trained on a large corpus and then fine-tuned on a specific dataset. We conducted extensive experiments and the results show that the pre-trained neural topic models significantly outperform baselines in terms of topic coherence and topic diversity. The proposed approach greatly reduces the amount of training data needed. In our experiments on the NYTimes dataset, a pre-trained model fine-tuned with 1% of documents achieves superior performance than baselines that are trained on the whole dataset. Due to the flexible modeling choices and high representation capacity, neural networks have been widely used for topic modeling in recent years. Some approaches (Kingma and Welling, 2013; Miao et al., 2016) model topics with variational autoencoders (VAEs) and view the latent variables of VAEs as document topics. However, topic models typically use Dirichlet distribution as the prior of multinomial topic distributions, while the repa-rameterization trick required by VAEs hinders the usage of a Dirichlet prior. Therefore, some followup works (Srivastava and Sutton, 2017; Card et al., 2018) used logistic normal to approximate Dirichlet. Another family of neural topic models (Nan et al., 2019; Wang et al., 2020; Hu et al., 2020) overcome the problem with adversarial training (Goodfellow et al., 2014) by encouraging the model to generate topic distributions that are similar to samples randomly drawn from a Dirichlet prior. There are mainly two ways to incorporate external knowledge into topic modeling, namely by PWEs and PLMs. Some attempts incorporate pre-trained word representations into neural topic models. For example, (Card et al., 2018; Dieng et al., 2020) used PWEs to initialize word embeddings of topic models. (Wang et al., 2020) built a generative process that models word embeddings with per-topic Gaussian distributions. Beyond static word embeddings, researchers also tried to utilize PLMs. (Bianchi et al., 2020b,a) treated PLM outputs as an additional knowledge source to enhance or replace BoW-based inputs. (Hoyle et al., 2020) employed knowledge distillation to guide the training of a student topic model with a PLM teacher network. Recently, (Song et al., 2020) proposed TopicOcean to train LDA-based topic models on large corpora and then transfer the knowledge of accumulated topics to new corpora which can also be considered a way of pre-training. It should be pointed out that the proposed PT-NTM differs from the previous PLMs-based topic models or TopicOcean in that the architecture of neural topic models during pre-training and fine-tuning are the same in PT-NTM while other methods combine the large PLM with the topic models, the two different model architectures. In this section, we describe the detailed processes of PT-NTM. First, we will introduce the architecture of neural topic model, which we call NTM in the following, employed in PT-NTM. Then, we will introduce how to pre-train the neural topic model on a large-scale dataset. Finally, we will introduce how to fine-tune the pre-trained neural topic model on the target dataset. For the architecture of NTM, we follow the encoder-decoder architecture, as employed by many neural topic models (Srivastava and Sutton, 2017; Miao et al., 2017; Nan et al., 2019). The encoder takes a document's BoW x  RV as input and infers its topic distribution  z  RK , where V is the vocabulary size and K the topic number. The decoder then reconstructs the original document from  z , denoted as  x . The whole architecture of NTM is shown in Figure 1. In specific, the encoder is a stack of N + 1 MLP layers. From the bottom to the top, the first N layers have an identical structure. Each layer has four sub-layers: Dropout (Srivastava et al., 2014), Linear, BatchNorm (Ioffe and Szegedy, 2015), and LeakyReLU (Maas et al., 2013). The final layer is a Dropout sub-layer and a Linear transformation followed by a Softmax. The decoder shares the same architecture as the encoder, though they may vary in input/output dimensions. In our experiments, we set a Dropout probability of 0 . 5 in the first encoder layer and 0 . 2 in the remaining encoder and decoder layers. All LeakyReLU sub-layers have a negative slope of 0 . 01 . which encourages the decoder outputs  X = {  x ( i ) } mi =1 to be as similar as the corresponding encoder inputs X = { x ( i ) } mi =1 for each training batch, where m is the batch size. For topic distribution  z , what we have done above is insufficient to generate reasonable topics since  z 's distribution Q is not well defined. To this end, we follow a similar approach proposed in (Nan et al., 2019) and further impose on  z a Dirichlet prior P by minimizing the Maximum Mean Discrepancy (MMD) (Gretton et al., 2012) between the two distributions P and Q : LMMD ( Z,  Z ) =  2 m 2 (cid:88) i,j k ( z ( i ) ,  z ( j ) )+ 1 m ( m  1) (cid:88) i  = j ( k ( z ( i ) , z ( j ) ) + k ( z ( i ) ,  z ( j ) )) , (2) where Z = { z ( i ) } mi =1 are topic distributions randomly drawn from the prior P ,  Z = {  z ( i ) } mi =1 are encoder outputs, and k is the kernel function that is information diffusion kernel (Lebanon and Lafferty, 2003) in our experiments following (Nan et al., 2019). where we balance L rec and LMMD with a hyperpa-rameter  and another factor where  2 denotes L2 normalization and b ( N +1) is the bias term of the last Linear sub-layer of the encoder, i.e., the one just before the Softmax sublayer. Equation (4) shows that the two losses are balanced with their relative gradient norm with respect to b ( N +1) . We found in our experiments that r greatly reduces the effort of tuning  and generally produces better results. By pre-training the topic model on a large and topically diverse corpus, we expect the model would learn topic-related knowledge that is general enough to be reused on other corpora. For the proposed approach, the knowledge may include word semantics, common senses, and document encoding and decoding patterns at each layer. The details of the pre-training procedure are presented in Algorithm 1. The pre-training corpus D is the subset00 of the OpenWebText dataset (Gokaslan and Cohen, 2019), an open-source recreation of the WebText dataset as detailed in (Radford et al., 2019). We preprocess data by tokenization, lemmatization, stopword removal, and only keeping words occurred in at least 50 documents. After preprocessing, there are about 392K documents, consisting of 45K unique words, in the resulting dataset. At each training mini-batch, we update model parameters according to Equation (3) using the Adam optimizer (Kingma and Ba, 2014). Algorithm 1 Pre-training. Require: D , the pre-training corpus; E , the encoder; D , the decoder;  , parameters of E and D ;  0 , initial parameters; m , the batch size; n , the number of training epochs; P ( z ) , the Dirichlet prior. 1:    0 2: for i = 1 ,    , n do 3: Shuffle D . 4: for each X = { x ( j ) } mj =1 from D do 5:  Z  E ( X ) ;  X  D (  Z ) 6: Sample Z = { z ( j ) } mj =1  P ( z ) . 7: Compute L by Equation (3). 8:   Adam (   1 m (cid:80) mj =1 L ( j ) ,  ) 9: end for 10: end for 3.3 Fine-tuning Fine-tuning is the process of adapting the pre-trained topic model to a specific dataset. However, directly fine-tuning the pre-trained model on a new dataset does not always work and may introduce severe bias to subsequent tuning steps since the ideal number of topics might change and the corpus-wide topic distributions might be different. Therefore, our fine-tuning begins with the pre-trained model but randomly re-initializes parameters in the last encoder layer and the first decoder layer. If we fine-tune the model without any re-initialization, we find that in our experiments the corpus-wide topic distributions discovered by the fine-tuned model would be biased towards the topic distribution of the pre-training corpus, which is unexpected. The proposed fine-tuning strategy with re-initialization solves this issue. Algorithm 2 shows the fine-tuning steps. We keep the pre-trained parameters fixed for the first n 1 epochs and use a small learning rate in the remaining training epochs since they have already been well trained before fine-tuning. Require: D  , the target corpus; E , the encoder; D , the decoder;  r , randomly initialized parameters;  p , pre-trained parameters; m , the batch size; n , the number of training epochs; n 1 , n 1  N and 0  n 1  n ; P ( z ) , the Dirichlet prior. 1: for i = 1 ,    , n do 2: Shuffle D  . 3: for each X = { x ( j ) } mj =1 from D  do 4:  Z  E ( X ) ;  X  D (  Z ) 5: Sample Z = { z ( j ) } mj =1  P ( z ) . 6: Compute L by Equation (3). 7:  r  Adam (   r 1 m (cid:80) mj =1 L ( j ) ,  r ) 8: if i > n 1 then 9:  p  Adam (   p 1 m (cid:80) mj =1 L ( j ) ,  p ) 10: end if 11: end for 12: end for By comparing Algorithm 1 with Algorithm 2, it can be observed that the fine-tuning process adds little overhead to the training stage. More importantly, the proposed method does not introduce any additional computations or parameters during inference. We used three datasets in (Hu et al., 2020): NYTimes 2 , Grolier 3 , and 20Newsgroups 4 . We did not include the DBPedia dataset as it is based on Wikipedia and potentially overlaps with the dataset used for our pre-training. The dataset statistics are shown in Table 1. The proposed basic model, NTM, is the one described in Section 3 without pre-training. Both the 2 http://archive.ics.uci.edu/ml/ datasets/Bag+of+Words 3 https://cs.nyu.edu/~roweis/data 4 http://qwone.com/~jason/20Newsgroups 5983 Dataset #Documents Vocabulary Size NYTimes 99,992 12,604 Grolier 29,762 15,276 20Newsgroups 11,258 2,000 Table 1: Dataset statistics. encoder and the decoder have three layers ( N = 2 ) and 300 neurons at each hidden layer. We have four variants:  NTM-w2v, we initialize weights w e 1  RV  300 of the first encoder Linear sub-layer and w d 3  R 300  V of the the last decoder Linear sub-layer with the corresponding 300 dim Word2Vec embeddings trained on Google News. NTM-glv, same as NTM-w2v but utilizing 300 -dim GloVe embeddings trained on Wikipedia and Gigaword 5.  PT-NTM-w2v, pre-training from NTM-w2v initialization and then fine-tuning. PT-NTM-glv, pre-training from NTM-glv initialization and then fine-tuning. The number of training epochs is 200 for pretraining, fine-tuning (PT-* models) and fresh training (NTM). We used the Dirichlet prior distribution whose parameters are all 1 K , where K is the topic number. MMD loss weight  is 1 for all models expect the fine-tuning of *-pre models in which  is 0.3. We will analyze the effect of  in our experiments. During pre-training, the batch size is 1,024, the learning rate is 2e-2, and the topic number is 200. For fine-tuning, n 1 is 100, and the learning rates for reinitialized and pre-trained parameters are 2e-2 and 1e-5, respectively (Algorithm 2), showing that the pre-trained parameters are only slightly tuned. The batch size of fine-tuning and fresh training varies on different datasets depending on their sizes. Specifically, it is set to 128 for 20Newsgroups, 256 for Grolier and 512 for NYTimes. Finally, it should be noted that fine-tuning on each datasets shares the same pre-trained model checkpoint for each model variant. LDA (Blei et al., 2003), we used the implementation of GibbsLDA++ 5 . ProdLDA (Srivastava and Sutton, 2017), a VAE-based model that employs logistic normal prior for topic distributions. W-LDA (Nan et al., 2019). Our model follows W-LDA loss but differs in training and implementation. BAT (Wang et al., 2020), an adversarially trained neural topic model. ToMCAT (Hu et al., 2020), an adversarial neural topic model with cycle-consistency objective. ZeroShotTM (Bianchi et al., 2020b), taking Sentence-BERT (Reimers and Gurevych, 2019) embeddings as input. CombinedTM (Bianchi et al., 2020a), same as ZeroShotTM but combining the input with BoWs. G-BAT (Wang et al., 2020), extending BAT to incorporate pre-trained word embeddings. TopicOcean (Song et al., 2020), integrating well-trained LDAs and transferring the knowledge of accumulated topics to new corpora, which is re-implemented by ourselves. We evaluate the model performance with three topic coherence measures and one topic diversity measure. Topic coherence measures first calculate the coherence scores of pairs of top words ranked by their topic-associated probabilities for each topic and then aggregate all topic scores as the final topic coherence. The used topic coherence measures are C_A (Aletras and Stevenson, 2013), C_P (Rder et al., 2015), and NPMI (Aletras and Stevenson, 2013) of top-10 topic words, implemented in Palmetto (Rder et al., 2015) 6 . Topic coherence measures are highly correlated with human evaluation but have no penalizing mechanism for repetitive or similar topics. We remedy the problem by also evaluating topic diversity. Our topic diversity measure is calculate by TD = 1  N rep N total , where N total = 10  K is the total number of topic words and N rep counts the number of repetitions in all topic words. For example, 5 identical words would add 4 to N rep . The topic modeling results are presented in Table 2. We report results averaged over five runs with topic number set to 20, 30, 50, 75, and 100 respectively in all our experiments unless otherwise specified. From Table 2, we can observe that: 1) Among all models, PT-NTM and its variants outperform other methods by a large margin. Since PT-NTM and NTM share the identical model architecture, we attribute the improvements of PT-NTM over NTM to the pre-training strategy. 2) For PLMs-based methods, both ZeroShotTM and CombinedTM performs badly, for some metric even worse than regular methods. We think the reason maybe the gap between the learning objectives of PLMs (word order-based) and topic models (word-cooccurrence based). 3) For PWEs-based methods, non-pretrained methods (NTM, BAT) benefits a lot from the PWEs. We think the reason maybe the PWEs are also trained based on word-cooccurrence, so the gap between PWEs and topic models is relatively small. Another interesting thing is that the benefit of using PWEs in topic modeling seems diminishing with our proposed topic model pretraining strategy. For example, PT-NTM gives similar results compared to PT-NTM-w2v and PT-NTM-glv. This shows that word semantic knowledge has somehow been captured to a certain degree by pre-training the topic model on a large corpus. 4) For pre-training-based models, PT-NTM outperforms TopicOcean, consider the performance gap between their base models (NTM for PT-NTM and LDA for TopicOcean), the improvement of PT-NTM is even larager. What's more, our method is based on neural network, which is easier to incorporated with PWEs or other information than TopicOcean, which is based on graphical models. One concern about PT-NTM may be that the whether the fine  tuning stage works. To get a sense of the topics extracted by our model, we list in Table 3 top 4 topics extracted by PT-NTM on the pre  training and fine  tuning dataset. The topic labels are assigned manually. The whole topics are presented in the attachment. Contextualized word embeddings like those produced by BERT (Devlin et al., 2019) provide richer semantic than static ones like Word2Vec (Mikolov et al., 2013) or GloVe (Pennington et al., 2014). Thus we also conducted experiments to test their performance on topic modeling. The baseline models are ZeroShotTM (Bianchi et al., 2020b) and CombinedTM (Bianchi et al., 2020a). ZeroShotTM and CombinedTM both take Sentence-BERT (Reimers and Gurevych, 2019) embeddings as inputs but CombinedTM additionally uses BoW. We also implement three NTM-based models, namely BERT-NTM, Word2Vec-NTM, and GloVe-NTM, according to the input embeddings they used. BERT-NTM follows the idea of ZeroShotTM, aim-5985 OpenWebText (Pre-training) NYTimes (Fine-tuning) Tesla Drug TPP GPU Racism Cuisine Health Weddding tesla marijuana tpp gtx racist shrimp fat wedding autonomous legalization nafta geforce racism sauce protein daughter waymo cannabi ustr nvidia trump cuisine calories bride driverless legalize trade amd black broth carbohydrate mother car norml freeland gpu feminist basil cup gown musk drug trump radeon political pork diet father vehicle dispensary tpa evga racial onion sugar wife autopilot decriminalization fta directx politic pastry chocolate husband automaker recreational mexico sli party garlic cholesterol sister hyperloop prohibition climate mhz women chef vitamin son Grolier (Fine-tuning) 20Newsgroups (Fine-tuning) Myth Artist History Biology Politics Terrorist Football Crime thor art emperor biology clinton bomb player police norse picasso empire organism president fbi game cop mythology artist justinian evolutionary bush fire team officer poseidon museum ottoman species tax waco nhl woman chariot sculpture byzantine physiology senate kill coach gun goddess painting throne gene political police defensive car athena exhibition king molecular secretary soldier season man god pollock roman fossil government military draft fbi sword portrait serbian genetic economy weapon winnipeg murder dragon monet war evolution administration terrorist league suspect Table 3: Top 4 topics extracted by PT-NTM on OpenWebText, NYTimes, Grolier and 20Newsgroups dataset. ing at providing a fair comparison between BERT-based topic models. Word2Vec-NTM only uses pre-trained embeddings in the encoder, which is different from NTM-w2v as the latter use the the pre-trained Word2Vec embeddings in both the first encoder layer and the last decoder layer. The same setup applies to GloVe-NTM. The experimental results on 20Newsgroups 7 are shown in Table 4. All the models have similar topic diversity. Our NTM variants outperform both ZeroShotTM and CombinedTM on all three topic coherence measures. The possible reasons could be: 1) Topic modeling does not quite rely on word order information, at least for our experimented dataset; and 2) Training of GloVe utilizes global word-word co-occurrence statistics that are also helpful for topic modeling. As topic modeling 7 The other two datasets only contain word counts, making it impossible to extract BERT embeddings since no word context information is present. can be viewed as a form of word clustering, our results are somewhat inline with previous findings reported in Meng et al. (2019) that using BERT leads to poor performance on text clustering. Number of model layers We vary the number of encoder and decoder layers of pre-training and fine-tuning models, and show the results in Table 5. It can be observed that the four-layer and the three-layer models achieve the highest topic coherence and topic diversity respectively. Further increasing the layer number resulted in slight declines in all four metrics. MMD loss weight  We present the impact of  on our model in Figure 2. With  increasing from 0.03 to 30, the NPMI of PT-NTM-glv first gradually increases, peaking at about 0.14 when  = 1 , and then gradually decreases. For Topic Diversity (TD), however, we observe a steady decline for 5986 0.03 0.1 0.3 1 3 10 30 0 0 . PT-NTM-glv. PT-NTM also has a similar trend but with more drastic changes. Given these findings, it seems that there is a trade-off towards generating more coherent or diverse topics. Nevertheless, it is worth noting that in comparison to NTM, the PT-NTM-glv is very robust to the choices of  . The NPMI values of PT-NTM-glv only fluctuate in the range of [0 . 11 , 0 . 14] while its TD values vary between 0.74 and 0.86. This is in contrast to NTM in which it has poor topic coherence for   0 . 1 and low topic diversity for   10 . We attribute the advantage of the pre-trained model to our proposed fine-tuning strategy. During fine-tuning, we mainly update a small set of parameters that are directly related to topics while only slightly tune others, which consequently enables more controllable data/gradient flows and thus produces more stable results. Data efficiency With pre-training, a topic model indeed captures extensive knowledge from an external corpus. As have been shown in our experiments, the acquired knowledge can improve the performance of subsequent fine-tuning on other datasets, It would be interesting to see to what extent such knowledge can increase data efficiency. To this end, we conducted experiments that take subsets of NYTimes dataset of varying sizes as training datasets. Specifically, we used dataset sizes including 1K, 2K, 4K,    , 64K, and 100K. For each size, we averaged the results over five runs whose training datasets are randomly sampled from the whole dataset with different random seeds. The results are shown in Figure 3. PT-NTM-glv has a very high starting point when the document number is 1000: the NPMI and TD is about 0.15 1 2 4 8 16 32 64 100 0 0 . and 0.89 respectively. While at the same time, NTM has extremely poor performance with negative NPMI and low TD. Only when the document number increases to 8000, the topics generated by NTM has comparable topic diversity to topics from PT-NTM-glv. But even when the whole dataset is used by PT-NTM, i.e., the document number is 100K, NTM's NPMI is still about 0.08 lower than the 1000-document PT-NTM-glv, which indeed represents a significant difference in topic quality. In summary, pre-training the topic model greatly reduces the need for training data and helps the model achieve superior performance with only 1% of documents on the NYTimes dataset. In this paper, we proposed a simple yet effective strategy to incorporating external knowledge into neural topic modeling by pre-training topic models on a large corpus before fine-tuning them on specific datasets. By experiments, we have presented the effectiveness of the method of pre-trained neural topic model in terms of topic coherence, topic diversity, and data efficiency over other methods such as by incorporating PWEs and PLMs. Another advantage of this approach is that it introduces little overhead to the training and none to the inference. Limited by computing resources, we did not experiment pre-trainings on larger datasets, though we believe there is still room for improvement given more pre-training data. For future research, we encourage further explorations in model architectures, pre-training objectives, and fine-tuning procedures. We would like to thank anonymous reviewers for their valuable comments and helpful suggestions and we thank Tencent for supporting this project. This work was funded by the National Natural Science Foundation of China (61772132, 62176053).
abstain result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain abstain abstain abstain abstain abstain abstain objective result objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain objective method abstain method objective other other