Even though BERT has achieved successful performance improvements in various supervised learning tasks, BERT is still limited by repetitive inferences on unsupervised tasks for the computation of contextual language representations. To resolve this limitation, we propose a novel deep bidirectional language model called a T ransformer-based T ext A utoencoder ( T-TA ). The T-TA computes contextual language representations without repetition and displays the benefits of a deep bidirectional architecture, such as that of BERT. In computation time experiments in a CPU environment, the proposed T-TA performs over six times faster than the BERT-like model on a reranking task and twelve times faster on a semantic similarity task. Furthermore, the T-TA shows competitive or even better accuracies than those of BERT on the above tasks. Code is available at https://github.com/joongbo/tta. A language model is an essential component of many natural language processing (NLP) applications ranging from automatic speech recognition (ASR) (Chan et al., 2016; Panayotov et al., 2015) to neural machine translation (NMT) (Sutskever et al., 2014; Sennrich et al., 2016; Vaswani et al., 2017). Recently, the Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019) and its variations have led to significant improvements in learning natural language representation and have achieved state-of-the-art performances on various downstream tasks such as the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) and question answering (Rajpurkar et al., 2016). BERT continues to succeed in various unsupervised tasks, such as the N -best list reranking for ASR and NMT (Shin et al., 2019; Salazar et al., 2019), con-firming that deep bidirectional language models are useful in unsupervised applications as well. However, concerning its applications to unsupervised learning tasks, BERT is significantly inefficient at computing language representations at the inference stage (Salazar et al., 2019). During training, BERT adopts the masked language modeling (MLM) objective, which is to predict the original word of the explicitly masked word from the input sequence. Following the MLM objective, each contextual word representation should be computed by a two-step process: masking a word in the input and feeding the result to BERT. During the inference stage, this process is repeated n times to obtain the representations of all the words within a text sequence (Wang and Cho, 2019; Shin et al., 2019; Salazar et al., 2019), resulting in a computational complexity of O ( n 3 ) 1 in terms of the number of words n . Hence, it is necessary to reduce the computational complexity when applying the model to situations where the inference time is critical, e.g. , mobile environments and real-time systems (Sanh et al., 2019; Lan et al., 2019). Considering this limitation of BERT, we submit a new research question: Can we construct a deep bidirectional language model with a minimal inference time while maintaining the accuracy of BERT? In this paper, in response to the above question, we propose a novel bidirectional language model named the T ransformer-based T ext A utoencoder ( T-TA ), which has a reduced computational complexity of O ( n 2 ) when applying the model to unsupervised applications. The proposed model is trained with a new learning objective named language autoencoding (LAE). The LAE objective, which allows the target labels to be the same as the text input, is to predict every token in the input sequence simultaneously without merely copying 1 A complexity of O ( n 2 ) is derived from the per-layer complexity of the Transformer (Vaswani et al., 2017). the input to the output. To learn the proposed objective, we devise both a diagonal masking operation and an input isolation mechanism inside the T-TA based on the Transformer encoder (Vaswani et al., 2017). These components enable the proposed T-TA to compute contextualized language representations at once while maintaining the benefits of the deep bidirectional architecture of BERT. We conduct a series of experiments on two unsupervised tasks: N -best list reranking and unsupervised semantic textual similarity. First, by conducting runtime experiments in a CPU environment, we show that the proposed T-TA is 6 . 35 times faster than the BERT-like model in the reranking task and 12 . 7 times faster in the unsupervised semantic textual similarity task. Second, despite its faster inference time, the T-TA achieves competitive performances relative to BERT on reranking tasks. Furthermore, the T-TA outperforms BERT by up to 8 points in Pearson's r on unsupervised semantic textual similarity tasks. When referring to an autoencoder for language modeling, sequence-to-sequence learning approaches have been commonly used. These approaches encode a given sentence into a compressed vector representation, followed by a decoder that reconstructs the original sentence from the sentence-level representation (Sutskever et al., 2014; Cho et al., 2014; Dai and Le, 2015). To the best of our knowledge, however, none of these approaches consider an autoencoder that encodes word-level representations (such as BERT) without an autoregressive decoding process. Many studies have been performed on neural network-based language models for word-level representations. Distributed word representations were proposed and attracted considerable interest, as they were considered to be fundamental building blocks for NLP tasks (Rumelhart et al., 1986; Bengio et al., 2003; Mikolov et al., 2013b). Subsequently, researchers explored contextualized representations of text where each word has a different representation depending on the context (Peters et al., 2018; Radford et al., 2018). Most recently, a Transformer-based deep bidirectional model was proposed and applied to various supervised-learning tasks with remarkable success (Devlin et al., 2019). For unsupervised tasks, researchers have adopted recently developed language-representation models and investigated their effectiveness; a typical example is the N -best list reranking for ASR and NMT tasks. In particular, studies have integrated left-to-right and right-to-left language models (Arisoy et al., 2015; Chen et al., 2017; Peris and Casacu-berta, 2015) to outperform conventional unidirectional language models (Mikolov et al., 2010; Sun-dermeyer et al., 2012) in these tasks. Furthermore, BERT-based approaches have been explored and have achieved significant performance improvements on these tasks because bidirectional language models yield the pseudo-log-likelihood of a given sentence, and this score is useful in ranking the n -best hypotheses (Wang and Cho, 2019; Shin et al., 2019; Salazar et al., 2019). Another line of research involves reducing the computation time and memory consumption of BERT. Lan et al. (2019) proposed parameter-reduction techniques, factorized embedding parameterization and cross-layer parameter sharing and reported 18 times fewer parameters and a 1 . 7 -fold increase in the training time. Similarly, Sanh et al. (2019) presented a method to pretrain a smaller model that can be fine-tuned for downstream tasks and achieved 1 . 4 times fewer parameters with a 1 . 6 fold increase in the inference time. However, none of these studies developed methods that directly revise the BERT architecture to reduce the computational complexity during the inference stage. In a conventional language modeling task, the i th token x i is predicted using its preceding context x <i = [ x 1 , . . . , x i  1 ] ; throughout this paper, this objective is known as causal language modeling (CLM) following (Conneau and Lample, 2019). As shown in Figure 1a, we can obtain (left-to-right) contextualized language representations HC = [ HC 1 , . . . , HC n ] after feeding the input sequence to the CLM-trained language model only once, where HC i = h C ( x <i ) is the hidden representation of the i -th token. This paper takes this unidirectional language model (uniLM) as our speed baseline. However, contextualized language representations obtained from the uniLM are insuf-ficient to accurately encode a given text because future contexts cannot be leveraged to understand the current tokens during the inference stage. Recently, BERT (Devlin et al., 2019) was designed to enable the full contextualization of language representations by using the MLM objective, in which some tokens from the input sequence are randomly masked; the objective is to predict the original tokens at the masked positions using only their context. As in Figure 1b, we can obtain a contextualized representation of the i -th token HM i = h M ( M i ( x )) by masking the token in the input sequence and feeding it to the MLM-trained model, where M i ( x ) = [ x 1 , . . . , x i  1 , [MASK] , x i +1 , . . . , x n ] signifies an external masking operation. This paper takes this bidirectional language model (biLM) as our performance baseline. However, this mask-and-predict approach should be repeated n times to obtain all the language representations HM = [ HM 1 , . . . , HM n ] because learning occurs only at the masked position during the MLM training stage. Although the resulting language representations are robust and accurate, as a consequence of this repetition, the model is significantly inefficient when applied to unsupervised tasks such as N -best list reranking (Wang and Cho, 2019; Shin et al., 2019; Salazar et al., 2019). In this paper, we propose a new learning objective named language autoencoding (LAE) for obtaining fully contextualized language representations without repetition. The LAE objective, with which the output is the same as the input, is to predict every token in a text sequence simultaneously without merely copying the input to the output. For the proposed task, a language model should reproduce the whole input at once while avoiding overfitting; otherwise, the model outputs only the representation copied from the input representation without learning any statistics of the language. To this end, the flow of information from the i -th input to the i -th output should be blocked inside the model shown in Figure 1c. From the LAE objective, we can obtain fully contextualized language representations HL = [ HL 1 , . . . , HL n ] all at once, where HL i = h L ( x \ i ) and x \ i = [ x 1 , . . . , x i  1 , x i +1 , . . . , x n ] . The method for blocking the flow of information is described in the next section. In this section, we introduce the novel architecture of the proposed T-TA shown in Figure 2. As indicated by its name, the T-TA architecture is based on the Transformer encoder (Vaswani et al., 2017). To learn the proposed LAE objective, we develop both a diagonal masking operation and an input isolation mechanism inside the T-TA. Both developments are designed to enable the language model to predict all tokens simultaneously while maintaining the deep bidirectional property (see the descriptions in the following subsections). For brevity, we refer to the original paper on the Transformer encoder (Vaswani et al., 2017) for other details regarding the standard functions, such as the multihead attention and scaled dot-product attention mechanisms, layer normalization, and the position-wise fully connected feed-forward network. As shown in Figure 3, a diagonal masking operation is implemented inside the scaled dot-product attention mechanism to be self-unknown during the inference stage. This operation prevents information from flowing to the same position in the next layer by masking out the diagonal values in the input of the softmax function. Specifically, the output vector at each position is the weighted sum of the value V at other positions, where the attention weights come from the query Q and the key K . The diagonal mask becomes meaningless when we use it together with a residual connection or utilize it within the multilayer architecture. To retain the self-unknown functional, we can remove the residual connection and adopt a single-layer architecture. However, it is essential to utilize a deep architecture to understand the intricate patterns of natural language. To this end, we further develop the architecture described in the next section. We now propose an input isolation mechanism to ensure that the residual connection and the multilayer architecture are compatible with the abovementioned diagonal masking operation. In the input isolation mechanism, the key and value inputs ( K and V , respectively) of all encoding layers are isolated from the network flow and are fixed to the sum of the token embeddings and the position embeddings. Hence, only the query inputs ( Q ) are updated across the layers during the inference stage by referring to the fixed output of the embedding layer. Additionally, we input the position embeddings to the Q of the very first encoding layer, thereby making the self-attention mechanism effective. Otherwise, the attention weights will be the same at all positions, and thus, the first self-attention mechanism will function as a simple average of all the input representations (except the self position). Finally, we apply the residual connection only to the query to completely maintain unawareness. The dashed arrows in Figure 2 show the proposed input isolation mechanism inside the T-TA. By using diagonal masking and input isolation in conjunction, the T-TA can have multiple encoder layers, enabling the T-TA to obtain high-quality contextual language representations after feeding a sequence into the model only once. Heretofore, we have introduced the new learning objective named LAE, and the novel deep bidirectional language model named T-TA. We will verify the architecture of the proposed T-TA in Section 4.3.1 and compare our model with the recently proposed strong baseline BERT in Section 4.3.2. Here, we discuss how diagonal masking with input isolation preserves the self-unknown property in detail. As shown in Figure 2, we have two input embeddings, namely, token embeddings X = [ X 1 , . . . , X n ] T  R n  d and position embeddings P = [ P 1 , . . . , P n ] T  R n  d , where d is an embedding dimension. From the input isolation mechanism, the key and value K = V = X + P have the information of the input tokens and are fixed in all layers, but the query Q l is updated across the layers during the inference stage starting from the position embeddings Q 1 = P in the first layer. Let us consider the l -th encoding layer's query input Q l and its output H l = Q l +1 : H l = SMSAN ( Q l , K , V ) = g ( Norm ( Add ( Q l , f ( Q l , K , V )))) , (1) where SMSAN (  ) is the self-masked self-attention network, namely, the encoding layer of the T-TA, g ( x ) = Norm ( Add ( x, FeedForward ( x ))) signifies two upper subboxes of the encoding layer in Figure 2, and f (  ) is the (multihead) diagonal-masked self-attention (DMSA) mechanism. As illustrated in Figure 3, the DMSA module computes Z l as follows: Z l = f ( Q l , K , V ) = DMSA ( Q l , K , V ) = SoftMax ( DiagMask ( Q l KT /  d )) V . In the DMSA module, the i -th element of Z l = [ Z l 1 , . . . , Z ln ] T is always computed by a weighted average of the fixed V while discarding the information of the i -th token X i in V i . Specifically, Z li is the weighted average of V with the attention weight vector s li , i.e. , Z li = s li V , where s li = [ s l 1 , . . . , s li  1 , 0 , s li +1 , . . . , s ln ]  R 1  n . Here, we note that the DMSA mechanism is related only to the self-unknown property since no token representations are referred to each other in subsequent transformations from Z l to H l . Therefore, we can guarantee that the i -th element of the query representation in any layer, Q li , never encounters the corresponding token representation starting from Q 1 i = P i . Consequently, the T-TA preserves the self-unknown property during the inference stage while maintaining the residual connection and multilayer architecture. There are several differences between the strong baseline BERT (Devlin et al., 2019) and the proposed T-TA, while both models learn deep bidirectional language representations. While BERT uses an external masking operation in the input, the T-TA has an internal masking operation in the model, as we intend. Additionally, while BERT is based on a denoising autoencoder, the T-TA is based on an autoencoder. With this novel approach, the T-TA does not need mask-and-predict repetition during the computing of contextual language representations. Consequently, we reduce the computational complexity from O ( n 3 ) with the BERT to O ( n 2 ) with the T-TA in applications to unsupervised learning tasks. As in the T-TA, feeding an intact input (without masking) into BERT is also possible. However, we argue that this process will significantly diminish the model performance in unsupervised applications since the MLM objective does not consider intact tokens much. In the next section, we include experiments that reveal the model performance with intact inputs (described in Tables 1, 3, and 4). For further reference, we also suggest a previous study that reported the same opinion (Salazar et al., 2019). To evaluate the proposed method, we conduct a series of experiments. We first evaluate the contextual language representations obtained from the T-TA on N -best list reranking tasks. We then apply our method to unsupervised semantic textual similarity (STS) tasks. The following sections will demonstrate that the proposed model is much faster than BERT during the inference stage (Section 5.2) while showing competitive or even better accuracies than those of BERT on reranking tasks (Sec-tion 5.3) and STS tasks (Section 5.4). The main purpose of this paper is to compare the proposed T-TA with a biLM trained with the MLM objective. For a fair comparison, each model has the same number of parameters based on the Transformer as follows: | L | = 3 self-attention layers with d = 512 input and output dimensions, h = 8 attention heads, and d f = 2048 hidden units for the position-wise feed-forward layers. We use a Gaussian error linear unit ( gelu ) activation function (Hendrycks and Gimpel, 2016) rather than the standard rectified linear unit ( relu ) following Ope-nAI GPT (Radford et al., 2018) and BERT (Devlin et al., 2019). In our experiments, we set the position embeddings to be trainable following BERT (Devlin et al., 2019) rather than a fixed sinusoid (Vaswani et al., 2017) with supported sequence lengths up to 128 tokens. We use WordPiece embeddings (Wu et al., 2016) with a vocabulary of approximately | V | (cid:39) 30 , 000 tokens. The weights of the embedding layer and the last softmax layer of the Transformer are shared. For the speed baseline, we also implement a uniLM that has the same number of parameters as the T-TA and biLM. For training, we create a training instance consisting of a single sentence with [BOS] and [EOS] tokens at the beginning and end of each sentence, respectively. We use 64 sentences as the training batch and train the language models over 1 M steps for ASR and 2 M steps for NMT. We train the language models with Adam (Kingma and Ba, 2014) with an initial learning rate of 1 e  4 and coefficients of  1 = 0 . 9 of  2 = 0 . 999 ; the learning rate is set to warm up over the first 50 k steps, and the learning rate exhibits linear decay. We use a dropout probability of 0.1 on all layers. Our implementation is based on Google's official code for BERT 2 . To train the language models that we implement, we use an English Wikipedia dump (approximately 13 GB in size) containing approximately 120 M sentences. The trained models are used for reranking in NMT and unsupervised STS tasks. For the ASR reranking task, we use additional in-domain training data, namely, 4.0 GB of normalized text data from the official LibriSpeech corpus containing approximately 40 M sentences. We first measure the runtime of each language model to compute the contextual language representation HL  R n  d of a given text sequence. In the unsupervised STS tasks, we directly use HL for the analysis. In the case of the reranking task, further computation is required: we compute Softmax ( HLET ) to obtain the likelihood of each token, where E  R | V | d is the weight parameter of the softmax layer. Therefore, the computational complexity of the reranking task is larger than that of the STS task. To measure the runtime, we use an Intel(R) Core(TM) i7-6850K CPU (3.60 GHz) and the Ten-sorFlow 1.12.0 library with Python 3.6.8 on Ubuntu 16.04.06 LTS. In each experiment, we measure the runtime 50 times and average the results. Figure 4 shows that the T-TA exhibits faster runtimes than the biLM, and the gap between the T-TA and biLM increases as the sentence becomes longer. To facilitate a numerical comparison, we set the standard number of words to 20 , which is approxi-2 https://github.com/google-research/bert Figure 4: Average runtimes of each model according to the number of words on STS and reranking tasks, subscripted as sts and rrk , respectively. mately the average number of words in a contemporary English sentence (DuBay, 2006). In this setup, in the STS tasks, the T-TA takes approximately 9 . 85 ms, while the biLM takes approximately 125 ms; hence, the T-TA is 12 . 7 times faster than the biLM. In the reranking task, the T-TA is 6 . 35 times faster than the biLM (which is still significant); this reduction occurs because the repetition of the biLM is related only to computing HL rather than Softmax ( HLET ) . For the visual clarity of Figure 4, we omit the runtime results of the uniLM, which is as fast as the T-TA (see Appendix B.1). With such a fast inference time, we next demonstrate that the T-TA is as accurate as BERT. To evaluate the language models, we conduct experiments on the unsupervised task of reranking the N -best list. In these experiments, we apply each language model to rerank the 50 best candidate sentences, which are obtained in advance using each sequence-to-sequence model on ASR and NMT. The ASR and NMT models we implement are detailed in Appendices A.1 and A.2. We rescore the sentences by linearly interpolating two scores from a sequence-to-sequence model and each language model as follows: score = (1   )  score s 2 s +   score lm , where score s 2 s is the score from the sequence-to-sequence model, score lm is the score from the language model calculated by the sum (or mean) of the log-likelihood of each token, and the interpolation weight  is set to a value that leads to the best performance in the development set. One of the strong baseline language models, the pretrained BERT-base-uncased model (De-vlin et al., 2019), is used for reranking tasks. We also include the reranking results from the traditional count-based 5 -gram language models trained on each dataset using the KenLM library (Heafield, 2011). We note that the T-TA and biLM (including BERT) assign the pseudo-log-likelihood to the score of a given sentence, whereas the uniLM assigns the log-likelihood. Because the reranking task is based on the relative scores of the n -best hypotheses, the fact that the bidirectional models yields the pseudo-log-likelihood of a given sentence does not impact this task (Wang and Cho, 2019; Shin et al., 2019; Salazar et al., 2019). For reranking in ASR, we use prepared N -best lists obtained from dev and test sets using Seq2Seq ASR , which we train on the LibriSpeech ASR corpus. Additionally, we use the N -best lists obtained from (Shin et al., 2019) to confirm the robustness of the language models in a testing environment. Table 1 shows the word error rates (WERs) for each method after reranking. The interpolation weights  are 0.3 or 0.4 in all N -best lists for ASR. First, we confirm that the bidirectional models trained with the LAE (T-TA) and MLM (biLM) objectives consistently outperform the uniLM trained with the CLM objective. The performance gains from reranking are much lower in the better base system Seq2Seq ASR , and it is evidently challenging to rerank the N -best list using a language model if the speech recognition model performs well enough. Interestingly, the T-TA is competitive with (or even better than) the biLM; this may result from the gap between the training and testing of the biLM: the biLM predicts multiple masks at a time when training but predicts only one mask at a time when testing. Moreover, the 3-layer T-TA is better than the 12-layer BERT-base, showing that in-domain data are critical to language model applications. Finally, we note that feeding an intact input to BERT (the corresponding model is denoted as w/ BERT \ M  in Table 1) causes the model to underper-form relative to the other models, demonstrating that the mask-and-predict approach is necessary for effective reranking. To compare the reranking performances in another domain, NMT, we again prepare N -best lists using Seq2Seq NMT 3 from the WMT13 German-to-English (De  En) and French-to-English (Fr  En) test sets. Table 2 shows the bilingual evaluation understudy (BLEU) scores for each method after reranking. Each interpolation weight becomes a value that shows the best performance on each test set with each method in NMT. The interpolation weights  are 0.4 or 0.5 in the N -best lists for NMT. We confirm again that the bidirectional models trained with the LAE and MLM objectives perform better than the uniLM trained with the CLM objective. Additionally, the Fr  En translation has less effect on the reranking than the De  En translation because the base NMT system for Fr  En is better than that for De  En. The 12 -layer BERT model appears much better than the other models at reranking on NMT; hence, the N -best hypotheses of the NMT model seem to be more indistinguish-3 The Seq2Seq models for De  En and Fr  En are trained independently using the t2t library (Vaswani et al., 2018). able than those of the ASR model from a language modeling perspective. All the reranking results on the ASR and NMT tasks demonstrate that the proposed T-TA performs both efficiently (similar to the uniLM) and effectively (similar to the biLM). In addition to the reranking task, we apply the language models to an STS task, that is, measuring the similarity between the meaning of sentence pairs. We use the STS Benchmark (STS-B) (Cer et al., 2017) and Sentences Involving Compositional Knowledge (SICK) (Marelli et al., 2014) datasets, both of which have a set of sentence pairs with corresponding similarity scores. The evaluation metric of STS is Pearson's r between the predicted similarity scores and the reference scores of the given sentence pairs. In this section, we address the unsupervised STS task to examine the inherent ability of each language model to obtain contextual language representations, and we mainly compare the language models that are trained on the English Wikipedia dump. To compute the similarity score of a given sentence pair, we use the cosine similarity of two sentence representations, where each representation is obtained by averaging each language model's contextual representations. Specifically, the contextual representations of a given sentence are the outputs of the final encoding layer of each model, denoted as context in Tables 3 and 4. For comparison, we use noncontextual representations, which are obtained from the outputs of the embedding layer, denoted as embed in Tables 3 and 4. As a strong baseline for unsupervised STS tasks, we also include the 12-layer BERT model (Devlin Method STS-B-dev STS-B-test context embed context embed BERT 64.78 -54.22 BERT \ M 59.17 60.07 47.91 48.19 BERT [CLS] 29.16 17.18 uniLM 56.25 63.87 39.57 55.00 uniLM [EOS] 40.75 38.30 biLM 59.99 -50.76 biLM \ M 53.20 58.80 36.51 49.08 T-TA 71.88 54.75 62.27 44.74 GloVe -52.4 -40.6 Word2Vec -70.0 -56.5 Table 3: Pearson's r  100 results on the STS-B dataset. et al., 2019), and we employ BERT in the mask-and-predict approach for computing the contextual representations of each sentence. Note that we use the most straightforward approach for the unsupervised STS task to focus on comparing token-level language representations. The STS-B dataset has 5749/1500/1379 sentence pairs with train/dev/test splits and corresponding scores ranging from 0 to 5. We test the language models on the STS-B-dev and STS-B-test sets using the simplest approach on the unsupervised STS task. As additional baselines, we include the results of GloVe (Pennington et al., 2014) and Word2Vec (Mikolov et al., 2013a) from the official sites of STS Benchmark 4 . Table 3 shows our T-TA trained with the LAE objective best captures the semantics of a sentence over the Transformer-based language models. Remarkably, our 3-layer T-TA trained on a relatively small dataset outperforms the 12-layer BERT trained on a larger dataset (Wikipedia + BookCor-pus). Furthermore, the embedding representations are trained better by the CLM objective than by the other language modeling objectives; we suppose that the uniLM depends strongly on the embedding layer due to its unidirectional context constraint. Since the uniLM encodes all contexts in the last token, [EOS] , we also use the last representation as the sentence representation; however, this approach does not outperform the average sentence 4 http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark Method SICK-test context embed BERT 64.31 BERT \ M 61.18 64.63 uniLM 54.20 65.69 biLM 58.98 biLM \ M 53.79 62.67 T-TA 69.49 60.77 Table 4: Pearson's r  100 results on the SICK dataset. representation. Similarly, BERT has a special token, [CLS] , which is trained for the next sentence prediction objective; thus, we also use the [CLS] token to see how this model learns the sentence representation, but it significantly underperforms the other models. We further evaluate the language models on the SICK dataset, which consists of 4934/4906 sentence pairs with training/testing splits and scores ranging from 1 to 5. The results are in Table 4, from which we obtain the same observations as those reported for STS-B. All results on unsupervised STS tasks demonstrate that the T-TA learns textual semantics best using the token-level LAE objective. In this work, we propose a novel deep bidirectional language model, namely, the T-TA, to eliminate the computational overload of applying BERT to unsupervised applications. Experimental results on N -best list reranking and unsupervised STS tasks demonstrate that the proposed T-TA is significantly faster than the BERT-like approach, and its encoding ability is competitive with (or even better than) that of BERT. K. Jung is with ASRI, Seoul National University, Korea. This work was supported by the Ministry of Trade, Industry & Energy (MOTIE, Korea) under the Industrial Technology Innovation Program (No.10073144) and by the NRF grant funded by the Korean government (MSIT) (NRF2016M3C4A7952587).
abstain objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain abstain abstain abstain abstain method objective abstain abstain abstain abstain other other method other other other other other other other other other other other other other other abstain abstain method abstain abstain abstain other method abstain abstain method abstain abstain abstain method abstain method abstain method abstain method abstain abstain abstain method method abstain method method abstain abstain method abstain method abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method method method abstain abstain abstain abstain method abstain method method method abstain method method method method abstain abstain method method method abstain method abstain abstain result abstain abstain result abstain other abstain abstain abstain abstain abstain abstain result abstain abstain result abstain other abstain result abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result result abstain abstain abstain abstain result abstain result abstain other abstain abstain abstain result abstain objective abstain other other