Transformer-based language models (TLMs), such as BERT, ALBERT and GPT-3, have shown strong performance in a wide range of NLP tasks and currently dominate the field of NLP. However, many researchers wonder whether these models can maintain their dominance forever. Of course, we do not have answers now, but, as an attempt to find better neural architectures and training schemes, we pretrain a simple CNN using a GAN-style learning scheme and Wikipedia data, and then integrate it with standard TLMs. We show that on the GLUE tasks, the combination of our pretrained CNN with ALBERT outperforms the original ALBERT and achieves a similar performance to that of SOTA. Furthermore, on open-domain QA (Quasar-T and SearchQA), the combination of the CNN with ALBERT or RoBERTa achieved stronger performance than SOTA and the original TLMs. We hope that this work provides a hint for developing a novel strong network architecture along with its training scheme. Our source code and models are available at https://github.com/nict-wisdom/bertac. Transformer-based language models (TLMs) such as BERT (Devlin et al., 2019), ALBERT (Lan et al., 2020), and GPT-3 (Brown et al., 2020) have shown that large-scale self-supervised pretraining leads to strong performance on various NLP tasks. Many researchers have used TLMs for various downstream tasks, possibly as subcomponents of their methods, and/or they have focused on scaling up TLMs or improving their pretraining schemes. As a result, other architectures like Recurrent Neural Networks (RNN) (Hochreiter and Schmidhu-ber, 1997; Cho et al., 2014) and Convolutional Neural Networks (CNN) (LeCun et al., 1999) are fading away. In this work, we propose a method ! for improving TLMs by integrating a simple conventional CNN to them. We pretrained this CNN on Wikipedia using a Generative Adversarial Network (GAN) style training scheme ( Goodfellow et al., 2014), and then combined it with TLMs. Oh et al. (2019) similarly used GAN-style training to improve a QA model using a CNN, but their training scheme was applicable only to QA-specific datasets. On the other hand, similarly to TLM, our proposed method for training the CNN is independent of specific tasks. We show that the combination of this CNN with TLMs can achieve higher performance than that of the original TLMs on publicly available datasets for several distinct tasks. We hope that this gives an insight into how to develop novel strong network architectures and training schemes. We call our combination of a TLM and a CNNBERTAC (BERT-style TLM with an Adversarially pretrained Convolutional neural network). Its architecture is illustrated in Fig. 1. We do not impose any particular restriction on the TLM in BERTAC, so any TLM, ALBERT (Lan et al., We used the CNN to compute representations of a slightly modified version of the input given to a TLM. To integrate these representations with those of the TLM, we stacked on top of the TLM several layers of Transformers for Integrating External Representation (TIERs) , which are our modified version of normal transformers (Vaswani et al., 2017). A TIER has the same architecture as that of a normal transformer encoder except for its attention: we replace the transformer's self-attention with an attention based on the representation provided by the CNN. We expect that, by keeping the basic architecture of transformer encoders, the CNN's representations can be integrated more effectively with the TLM's original representations. We pretrained the CNN using a GAN-style training scheme in order to generate representations of sentences rather freely without the constraint of token embedding prediction in the masked language modeling used for TLMs, as we explain later. For the training, we used masked sentences autogenerated from Wikipedia. As in the masked language modeling, neither human intervention nor downstream task-specific hacking is required. As illustrated in Fig. 2, the GAN-style training requires three networks, namely, a discriminator D and two CNN-based generators R and F . Once the training is done, we use the generator F as CNN in BERTAC. The training data consists of pairs of an entity mention and a sentence in which the entity mention is masked with a special token [EM] . For example, the entity-masked sentence m 1 in Table 1 is obtained by masking the entity mention e 1 , Suvarnabhumi Airport, in the original text s 1 . The network F generates a vector representation of the masked sentence ( m 1 ), while R produces a representation of the masked entity ( e 1 ). The discriminator D takes representations generated by either R or F as the input, and it predicts which generator actually gave the representation. In the original GAN, a generator learns to generate an artificial image from random noise so that the resulting artificial image is indistinguishable from given real images. By analogy, we used an entity-masked sentence as random noise and a masked entity as a real image. In our GAN-style training, we regard the vector representation of a masked entity given by generator R as a real representation of the entity (or the representation of the real image in the above analogy). On the other hand, we regard the representation of the masked sentence, generated by F , as a fake representation of the entity (or the representation of the artificial image generated from the random noise in the above analogy). This representation is deemed fake because the entity is masked in the masked sentence, and F does not know what the entity is exactly. During the training, F should try to deceive the discriminator D by mimicking the real representation and generating a fake representation that is indistinguishable from the real representation of the entity generated by R . On the other hand, R and D , as a team, try to avoid being mimicked by F and also to make the mimic problem harder for F . If everything goes well, once the training is over, F should be able to generate a fake representation of the entity that is similar to its real representation. An interesting point is that F 's output can be interpreted in two ways: it is a representation of a masked sentence because it is computed from the sentence, and at the same time it is a representation of the masked entity because it is indistinguishable from R 's representation of the entity. This duality suggests that F 's output can be seen as a representation of the entire sentence . 2105 We exploit F as a CNN in BERTAC as follows: first, we use F to compute a representation of a masked version of the sentence originally given as input to a TLM. The entity mention to be masked is chosen by simple rules and, if the input consists of multiple sentences, we generate a representation of each (masked) input sentence and concatenate these together into a single one. Then, this representation is integrated to the output of the TLM through multiple TIER layers. Our GAN-style pretraining is conceptually similar to TLM pretraining with masked language modeling (predicting what a masked word in a sentence should be). However, it was designed to pretrain a model that is able to rather freely generate entity representations without strongly sticking to the prediction of token embeddings. Our hypothesis is that such freely generated representations may be useful for improving the performance of downstream tasks. Moreover, we assumed that using multiple text representations computed from different perspectives (i.e., predicting token embeddings and freely generating entity representations) would help to improve the performance of downstream tasks. In our experiments, we show that for the GLUE tasks ( Wang et al., 2018), BERTAC's average performance on the development set was 0.7% higher than that of ALBERT, which was used as a subcomponent of BERTAC, leading to a performance on the test set comparable to that of SOTA (90.3% vs 90.8% (SOTA)). It also outperformed the SOTA method of open-domain QA ( Chen et al. , 2017) on Quasar-T (Dhingra et al., 2017) and SearchQA (Dunn et al., 2017) using either ALBERT or RoBERTa. We also compared our method with alternative models using a CNN pretrained in a self-supervised (non GAN-style) manner to directly predict embeddings of the entity mentions. Consequently, we confirmed that our method worked better: only the CNN trained by our GAN-style pretraining gave significant performance improvement over base TLMs. Note that the computational overhead of BERTAC is reasonably small. It took 20 hours with 16 GPUs to pretrain a single CNN model and 180 hours for the nine models tested with different parameter settings in this work (cf., 480 hours with 96 GPUs for pretraining DeBERTa (He et al., 2021), for example). Moreover, once pretrained, the CNN models can be re-used for various downstream tasks and combined with various TLMs, including potentially future ones. As for the parameter number, BERTAC had just a 14% increase in parameters when ALBERT-xxlarge was used as its base TLM (268 M parameters for BERTAC vs. 235 M for ALBERT-xxlarge). We confirmed from these results that BERTAC could improve pretrained TLMs with reasonably small computational overhead. The code and models of BERTAC are available at https://github.com/nict-wisdom/bertac. 2 Related Work Pretraining TLMs with entity information : There have been attempts to explicitly learn entity representation from text corpora using TLMs (He et al., 2020; Peters et al., 2019; Sun et al., 2020; Wang et al., 2020a; Xiong et al., 2020; Zhang et al., 2019). Our proposed method is a complementary alternative to these existing methods in the sense that entity representations are integrated into TLMs via CNNs and not directly produced by the TLMs. Fine-tuning TLMs with external resources or other NNs : Yang et al. (2019a) and Liu et al. (2020) have used knowledge graphs for augmenting TLMs with entity representations during fine-tuning. Unlike these approaches, BERTAC uses unstructured texts rather than clean structured knowledge, such as knowledge graphs, to adversarially train a CNN. Other previous works have proposed combining CNNs or RNNs with BERT for NLP tasks (Lu et al., 2020; Safaya et al., 2020; Shao et al., 2019; Zhang et al., 2020), but their use of CNNs/RNNs was task-specific, so their models were not directly applicable to other tasks. Adversarial learning for improving TLMs : Oh et al. (2019) proposed a CNN-based answer representation generator for QA that can guess the vector representation of answers from given why-type questions and answer passages. The generator was trained in a GAN-style manner using QA datasets. We took inspiration from their adversarial training scheme to train task-independent representation generators from unsupervised texts (i.e., Wikipedia sentences in which an entity was masked in a cloze-test style). ELECTRA (Clark et al., 2020) also employed an adversarial technique (not a GAN) to pretrain two TLMs: A generator was trained to perform masked language modeling and a discriminator 2106 was trained to distinguish tokens in the training data from tokens replaced by the generator. On downstream tasks, only the discriminator was fine-tuned. In BERTAC, the GAN-style pretraining was applied only to the CNN, thus reducing the training cost. Furthermore, the CNN can be combined easily with any available TLM, even potentially future ones, without having to re-do the pretraining. In this work, we show that BERTAC outperformed ELECTRA on the GLUE task. Vernikos et al. (2020) proposed a method that used an adversarial objective and an adversarial classifier for regularizing the fine-tuning process of TLMs, inspired by adversarial learning for domain adaptation (Ganin et al., 2016). Our work uses a GAN-style training scheme only for pretraining CNNs, not for fine-tuning TLMs. 3 Pretraining of CNNs This section describes the training data and training algorithm for our CNN. 3.1 Training data We pretrained our CNN with an entity-masked version of Wikipedia sentences. WikiExtractor 1 was used to extract, from the English Wikipedia 2 , sentences that have at least one entity mention, i.e., an entity with an internal Wikipedia link. Then we randomly selected one entity mention e i in each sentence and generated an entity-masked sentence m i by replacing the entire selected mention with [EM] . For example, we generated the masked sentence m 1 ,  [EM] is Thailand's main international air hub, (in Table 1) by replacing the entity mention e 1 , Suvarnabhumi Airport , in the sentence s 1 ,  ::::::::::::: Suvarnabhumi ::::::: Airport is Thailand's main international air hub, with [EM] . We obtained about 43.3 million pairs of an entity mention and a masked sentence ( f ( e i , m i ) g ) in this way and used 10% of them (randomly sampled) as the pretraining data for our CNN. 3.2 GAN-style pretraining As illustrated in Fig. 2, the adversarial training is done using three subnetworks: R ( real-entity-representation generator ), F ( fake-entity-representation generator ), and D ( discriminator ). R and F are CNNs with average pooling and D 1 https://github.com/samuelbroscheit/wikiextractor-wikimentions 2 We used the September 2020 version. is a feedforward neural network. Once the training is done, we use the generator F as CNN in BERTAC. In the training, we regard the representation of a masked entity output by generator R as a real representation of the entity that the fake-entity-representation generator F should mimic. F is trained so that, taking an entity-masked sentence as its input, it can generate a representation of the masked entity mention (called a fake representation of the entity in this work) that D cannot distinguish from the real representation. The representation generated by F is fake in the sense that the entity mention is masked in the input sentence and F cannot know what it is exactly. As mentioned in the Introduction, our GAN-style pretraining was designed to train a model capable of freely generating entity representations . We assumed that using multiple text representations computed from different perspectives (i.e., prediction of token embeddings in TLMs and generation of entity representations in our CNN) would help to improve the performance of downstream tasks. Algorithm 1: Adversarial Training Scheme Input: Training examples f ( e , m ) g , training epochs t , mini-batch steps b , mini-batch size n Output: Real representation generator R , fake representation generator F , discriminator D 1 j 1 2 Initialize (cid:18) R , (cid:18) F , and (cid:18) D (parameters of R , F , and D ) with random weights 3 while j (cid:20) t do 4 k 1 5 while k (cid:20) b do 6 Sample mini-batch of n examples f ( e i , m i ) g ni =1 7 Generate word embeddings f ( e i , m i ) g ni =1 of the examples. 8 Update D and R by ascending their stochastic gradient:  (cid:18) D ;(cid:18) R 1 n n  i =1 [log D ( R ( e i )) + log ( 1 (cid:0) D ( F ( m i )) ) ] 9 Update F by descending its stochastic gradient:  (cid:18) F 1 n n  i =1 log ( 1 (cid:0) D ( F ( m i )) ) 10 k k + 1 11 end 12 j j + 1 13 end For each pair of an entity mention ( e i ) and an entity-masked sentence ( m i ) in the training data, we first generate two matrices of word embeddings e i and m i using word embeddings pretrained on Wikipedia with fastText (Bojanowski et al., 2017). Then, R and F generate, respectively, a 2107 real entity representation from e i and a fake entity representation from m i . Finally, they are given to D , which is a feed-forward network that judges whether F or R generated the representations, i.e., whether the representations are real or fake , using sigmoid outputs by the final logistic regression layer. The pseudo code of the training scheme is given in Algorithm 1 . The training proceeds as follows: R and D as a team try to avoid the possibility that D misjudges F's output (i.e., a fake entity representation) as a real entity representation. More precisely, R and D are trained so that D can correctly judge the representation R ( e i ) given by generator R as real (i.e., D ( R ( e i )) = 1 ) and the representation F ( m i ) given by generator F as fake (i.e., D ( F ( m i )) = 0 ). Therefore, the training is carried out with the objective of maximizing log D ( R ( e i )) + log ( 1 (cid:0) D ( F ( m i )) ) (line 8 in Algorithm 1 ). On the other hand, F tries to generate representation F ( m i ) so that D judges it as real (i.e., D ( F ( m i )) = 1 ). Thus, F is trained to minimize log ( 1 (cid:0) D ( F ( m i )) ) (line 9 in Algorithm 1 ). This minmax game is iterated for the pre-specified t training epochs. 3.3 Pretraining settings We extracted 43.3 million pairs of an entity mention and a masked sentence from Wikipedia and randomly sampled 10% of them to use as training data (4.33 million pairs, around 700 MB in file size). We used word-embedding vectors in 300 dimensions (for 2.5 million words) pretrained on Wikipedia using fastText ( Bojanowski et al., 2017). The embedding vectors were fixed during the training. We set the training epochs to 200 ( t = 200 in Algorithm 1 ) and did not use any early-stopping technique. We chose t = 200 from the results of our preliminary experiments in which we used 10% of the training data and set training epochs t to either of 100, 200, or 300; the loss robustly converged for t = 200 and t = 300 , and thus the earliest point t = 200 was chosen. We used the Rm-sProp optimizer (Tieleman and Hinton, 2012) with a batch size of 4,000 ( n = 4 ; 000 and b = 1 ; 084 in Algorithm 1 ) and a learning rate of 2e-4. We trained nine CNN models with all combinations of the filter's window sizes 2 f 1,2,3, 2,3,4, 1,2,3,4 g and number of filters 2 f 100, 200, 300 g for the generators F and R . All of the weights in the CNNs were initialized using He's method (He et al., 2015). We used a logistic regression layer with sigmoid outputs as discriminator D . The training of a single CNN model took around 20 hours using 16 Nvidia V100 GPUs with 32 GB of memory (180 hours in total for the nine models). We tested all nine CNN models for BERTAC in our GLUE and open-domain QA experiments (Section 5). For each task, the parameters inside the CNNs (as well as the word-embedding vectors) were fixed during the fine-tuning of BERTAC. 4 BERTAC As illustrated in Fig. 1, BERTAC (BERT-style TLM with an Adversarially pretrained Convolutional neural network) incorporates the representation provided by the adversarially pretrained CNN to the representation generated by a TLM. For the integration, we use several layers of TIERs (Trans-formers for Integrating External Representation) stacked on top of the TLM. 4.1 CNN in BERTAC For simplicity, we describe how the CNN is integrated in BERTAC using the task of recognizing textual entailment (RTE) as an example. BERTAC for the RTE task takes two sentences s x and s y as input and predicts whether s x entails s y . First, we explain how the adversarially pretrained CNN (generator F in Section 3.2) generates the representation of the two input sentences. We regard the longest common noun phrase 3 of the two sentences as the entity mention to be masked and create entity-masked sentences m x and m y from s x and s y by masking the noun phrase with [EM] (we use m x = s x and m y = s y if no common noun phrase is found). Then each of the masked sentences m x and m y is given to the CNN. Our expectation here is that the CNN generates similar representations from the masked sentences if they have an entailment relation and that this helps to recognize the entailment relation. Note that the CNN in BERTAC is connected to several TIER layers and that, as shown in Fig. 1, its input is iteratively updated so that it provides updated representations to the TIER layers. Let m ix 2 R j m x j(cid:2) d w and m iy 2 R j m y j(cid:2) d w be the matrices of word embeddings of m x and m y given 3 For single-sentence tasks such as CoLA (Wang et al., 2018), we regard the longest noun phrase in a sentence as an entity. 2108 to the CNN connected to the i -th TIER layer, where d w is the dimension of a word embedding. We denote the representation generated by the CNN when the matrix of word embeddings m was used as the input by CNN ( m ) . The i th TIER layer is given the concatenation of the two CNN representations of m x and m y , r i = [ r ix ; r iy ] 2 R 2 (cid:2) d e , where r ix = CNN ( m ix ) 2 R d e , r iy = CNN ( m iy ) 2 R d e and d e is the dimension of the CNN representation. Note that, for single-sentence tasks, r i = r ix , the CNN representation of m x , is given to the TIER layers. The initial matrices of word embeddings m 1 x and m 1 y are obtained using the fastText word embeddings (Bojanowski et al., 2017), the same as that used in our adversarial learning. Then, the updated input matrices m i +1 x and m i +1 y for the ( i +1) th CNN are obtained from the i -th input matrices m ix and m iy as described below. For the word embedding m i x;j of the j -th word in m x , we compute its bilinear score to r ix ( Sutskever et al., 2009): (cid:22) m ix;j = softmax j ( m i T x B ix r ix ) m ix;j ; where B ix 2 R d w (cid:2) d e is a trainable matrix and softmax j ( v ) denotes the j -th element of the soft-maxed vector of v . The bilinear score indicates how much the corresponding token should be highlighted as one associated with the CNN representation r ix during the update process. We expect that this allows the CNN in the next TIER layer to generate further refined representations with the updated embeddings. We then compute word embeddings m i +1 x in a highway network manner ( Srivastava et al., 2015) as follows: m i +1 x = H x ( (cid:22) m ix )  T x ( m ix )+ m ix  (1 (cid:0) T x ( m ix )) ; where H x ( m ix ) = W ih m ix + b ih , T x ( m ix ) = (cid:27) ( W i t m ix + b i t ) , (cid:27) is the sigmoid function,  represents the element-wise product, and W ih , W it , b ih , and b it are layer-specific trainable parameters. m i +1 y is also computed from m iy and r iy in the same way. During the fine-tuning of BERTAC for downstream tasks, we fix the parameters of the pretrained CNN but train these parameters for updating CNN's input alongside those of TLMs and TIERs. 4.2 Transformers for integrating external representation (TIERs) As explained in the Introduction, the main difference between a TIER and a normal transformer In the TIER attention mechanism, the query representation, which is one of the three inputs of the transformer's self-attention, is replaced with the representation given by the CNN. Fig. 3 shows the difference between the TIERs' attention computation and that of normal transformers. Attention in normal transformers is computed in the following way: Attention ( Q ; K ; V ) = softmax ( QKT p d k ) V : Q , K , and V are query, key, and value matrices in R l k (cid:2) d k , where l k is the length of an input sequence and d k is a dimension of keys. Q , K , and V all come from the same representation of the token sequence provided from the previous transformer layer. The attention should specify how much the corresponding tokens in V should be highlighted, so we designed ours in the same way. In TIERs, we use the following attention. We basically replace the matrix Q with the CNN's representation r 2 R u (cid:2) d k while keeping the original K and V , where u is the number of sentences in the input of the model ( u 2 f 1 ; 2 g in this paper). Since r is a matrix with a different size from Q , we needed to adapt the attention computation. We first multiply r to KT , and then its soft-maxed results are converted into a l k (cid:2) d k dimensional matrix using the all-one matrix J u;d k 2 R u (cid:2) d k . Let the resulting matrix be A = ( softmax ( rK T p d k )) TJ u;d k 2 R l k (cid:2) d k . We apply the attention score to V by using the element-wise product between matrices: A  V . In addition, the actual CNN's representation r CNN 2 R u (cid:2) d e given by our CNNs usually have a size that does not match the size requirement for r . Thus, we convert it to r 2 R u (cid:2) d k , a d k column matrix as follows: r = r CNNW + b , where W 2 R d e (cid:2) d k and b are trainable. GLUE (Wang et al., 2018) is a multi-task benchmark composed of nine tasks including two single-sentence tasks (CoLA and SST-2) and seven two-sentence tasks of similarity/paraphrase tasks (MRPC, QQP, and STS-B) and natural language inference tasks (MNLI, QNLI, RTE, and WNLI). Following the previous work of ALBERT (Lan et al. , 2020), we performed single-task fine-tuning for each task under the following settings: single-model for the development set and ensemble for test set submissions. As in Liu et al. (2019) and Lan et al. (2020), we report the performance on the development set for each task by averaging over five runs with different random initialization seeds. As in Lan et al. (2020), for test set submissions, we fine-tuned the models for the RTE, STS-B, and MRPC tasks by initializing them with the fine-tuned MNLI single-task model, and we also used task-specific modification for CoLA and WNLI to improve scores (see Appendix A for de-tails). We explored ensemble settings between 6 and 30 models per task for our test set submission. We used ALBERT-xxlarge-v2 (Lan et al., 2020) as the pretrained TLM. As hyperparameters for BERTAC, for each task we tested learning rates 2 f 8e-6, 9e-6, 1e-5, 2e-5, 3e-5 g , a linear warmup for the first 6% of steps followed by a linear decay to 0, a maximum sequence length of 128, and all nine CNNs pretrained with different filter settings. We set the batch size to 128 for MNLI and QQP and 16 for the other tasks. Furthermore, we trained our model with the following set of training epochs: f 1,2,3,4,5 g for MNLI, QQP, and QNLI, f 6,7,8,9,10 g for CoLA, MRPC, RTE, SST-2, and STS-B, and f 90,95,100,105,110 g for WNLI. We set the number of TIER layers to 3 after preliminary experiments. See Table 9 in Appendix B for a summary of the hyperparameters tested in the GLUE experiments. During the fine-tuning of BERTAC, the parameters inside the CNNs (as well as word embeddings of fastText) were fixed as explained in Section 3.3, while those used to update the input to the CNNs were optimized. For each task, we selected the pretrained CNN (out of nine) and the BERTAC hyperparameters that gave the best performance on the development data. Table 2 shows the results of eight tasks on the GLUE development set: all of them are single-model results. Our BERTAC consistently outperformed the previous TLM-based models over seven tasks, except for QQP, and, as a result, showed the best average performance on the development set. Crucially, our model improved the average performance around 0.7% over ALBERT, the base TLM in our model. This indicates the effectiveness of adversarially trained CNNs and TIERs in BERTAC. The test set results obtained from the GLUE leaderboard are summarized in Table 3. Our model showed comparable performance to SOTA, DeBERTa/TuringNLRv4, and achieved state-of-the-art results on 3 out of 9 task. It also showed better performance than ALBERT, our base TLM, in most tasks. To investigate whether our GAN-style pretraining of CNNs contributed to the performance improvement, we also tested the following alternative training schemes for the CNN used in BERTAC. Self-supervised CNN: We pretrained the CNN to generate representations of a masked sentence in a self-supervised way as follows: For an entity mention e and an entity-masked sentence m in the training data (Section 3.1), the CNN generates a representation r from the masked sentence trying to minimize MSE (mean squared error) between r and the entity mention's representation e (average word embedding of all tokens in e ). Randomly initialized CNN: We did not pretrained the CNNs, but trained them alongside the TLMs during the fine-tuning of BERTAC (the CNNs were randomly initialized). We trained both the self-supervised and randomly initialized CNNs using the same hyperpa-rameter settings as GAN-style CNNs (see Section 3.3). We confirm from the results in Table 4 2110 Models MNLI QNLI QQP RTE SST-2 MRPC CoLA STS-B Avg. that only the proposed method with our GAN-style CNNs showed a higher average score than ALBERT. This suggests the effectiveness of our GAN-style pretraining scheme of CNNs. We also tested BERTAC on open-domain QA ( Chen et al., 2017) with the publicly available datasets Quasar-T (Dhingra et al., 2017) and SearchQA ( Dunn et al., 2017). We used the pre-processed version 4 of the datasets provided by Lin et al. (2018), which contains passages retrieved for all questions, and followed their data split as described in Table 5. We implemented our QA model following the approach of Lin et al. (2018), which combines a passage selector to choose relevant passages from retrieved passages and an answer span selector to identify the answer span in the selected passages. For the given question q and the set of retrieved passages P = f p i g , we computed the probability P r ( a j q; P ) of extracting answer span a to question q from P in the following way, and then we extracted the answer span ^ a with the highest probability: P r ( a j q; P ) =  i P r ( a j q; p i ) P r ( p i j q; P ) ; 4 Available at https://github.com/thunlp/OpenQA where P r ( p i j q; P ) and P r ( a j q; p i ) are computed by the passage selector and answer span selector, respectively. We input  [CLS] question [SEP] passage [SEP]  to both the passage selector and answer span selector, where [CLS] and [SEP] are special tokens. In the passage selector, the representation of [CLS] in the top TIER layer is fed into a linear layer with a softmax, which computes the probability that the passage contains a correct answer to the question. Our BERTAC answer span selector identifies answer spans from passages by computing start and end probabilities of each token in passages, where we feed the representation of each token in the top layer of TIERs to two linear layers, each with a softmax for the probabilities (Devlin et al., 2019). 5.2.2 Training details for open-domain QA We used all nine pretrained CNNs, as in the GLUE experiments. As pretrained TLMs, we used ALBERT-xxlarge-v2 (Lan et al., 2020) and RoBERTa-large (Liu et al., 2019). We set the learning rate to 1e-5, the number of epochs to 2, the maximum sequence length to 384, and the number of TIER layers to 3. We used a linear warmup for the first 6% of steps followed by a linear decay to 0 with a batch size of 48 for Quasar-T and 96 for SearchQA. We tested all of the pretrained CNNs and chose for each dataset the one that maximizes EM (the percentage of the predictions matching exactly one of the ground truth an-2111 CNNs used in BERTAC MNLI QNLI QQP RTE SST-2 MRPC CoLA STS-B Avg. See Table 10 in Appendix B for a summary of the hyperparameters tested for open-domain QA. We compared BERTAC with the previous works described in Table 6. Table 7 shows the performance of all of the methods. The subscripts of the TLM-based methods represent the type of pretrained TLM used by each method. All the methods were evaluated using EM and F1 score (average overlap between the prediction and gold answer). BERTAC ALBERT-xxlarge outperformed all of the baselines including the SOTA method (CFORMER ) on both EM and F1. BERTAC RoBERTa-large in the same TLM setting as the SOTA method showed a better performance than SOTA except for F1 in Quasar-T. These results suggest that our framework is effective for QA tasks as well. TIER, which uses ALBERT-xxlarge alone without using our CNN and TIER, w/o GAN-style CNN, which does not use our CNN pretrained by the GAN-style training scheme but uses self-supervised CNNs (the same as used in the GLUE experiments, see Table 4), w/o update, which does not perform layer-wise update of the CNN inputs. The results in Table 8 suggest that all of the following contributed to the performance improvement: the combination of TLMs and GAN-style CNNs, our GAN-style training of CNNs, and the layer-wise update of the CNN inputs. We proposed BERTAC (BERT-style TLM with an Adversarially pretrained Convolutional neural net-work), a combination of a TLM and a CNN, where the CNN was pretrained using a novel GAN-style training scheme and masked sentences obtained automatically from Wikipedia. Using this CNN, we improved the performance of standard TLMs. We confirmed that BERTAC could achieve comparable performance with the SOTA and outperformed the base TLM used as a subcomponent of BERTAC in the GLUE task. We also show that BERTAC outperformed the SOTA method of open-domain QA on Quasar-T and SearchQA.
abstain abstain result result abstain objective other abstain abstain abstain objective other abstain method method result result other abstain abstain abstain abstain result method method abstain objective abstain method abstain method abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain result result abstain other abstain result result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain objective result result result