One of the most crucial challenges in question answering (QA) is the scarcity of labeled data, since it is costly to obtain question-answer (QA) pairs for a target text domain with human annotation. An alternative approach to tackle the problem is to use automatically generated QA pairs from either the problem context or from large amount of unstructured texts (e.g. Wikipedia). In this work, we propose a hierarchical conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts, while maximizing the mutual information between generated QA pairs to ensure their consistency. We validate our Info rmation Maximizing H ierarchical C onditional V ariational A uto E ncoder ( Info-HCVAE ) on several benchmark datasets by evaluating the performance of the QA model (BERT-base) using only the generated QA pairs (QA-based evaluation) or by using both the generated and human-labeled pairs (semi-supervised learning) for training, against state-of-the-art baseline models. The results show that our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training. 1 1 Introduction Extractive Question Answering (QA) is one of the most fundamental and important tasks for natural language understanding. Thanks to the increased complexity of deep neural networks and use of knowledge transfer from the language models pretrained on large-scale corpora (Peters et al., 2018; Devlin et al., 2019; Dong et al., 2019), the state-of-the-art QA models have achieved human-level performance on several benchmark datasets (Ra-jpurkar et al., 2016, 2018). However, what is also * Equal contribution 1 The generated QA pairs and the code can be found at https://github.com/seanie12/Info-HCVAE Paragraph (Input) Philadelphia has more murals than any other u.s. city, thanks in part to the 1984 creation of the department of recreation's mural arts program, ...The program has funded more than 2,800 murals Q1 which city has more murals than any other city? A1 philadelphia Q2 why philadelphia has more murals? A2 the 1984 creation of the department of recreation's mural arts program Q3 when did the department of recreation' s mural arts program start ? A3 1984 Q4 how many murals funded the graffiti arts program by the department of recreation? A4 more than 2,800 Table 1 : An example of QA pairs generated with our framework. The paragraph is an extract from Wikipedia provided by Du and Cardie (2018). For more examples, please see Appendix D. crucial to the success of the recent data-driven models, is the availability of large-scale QA datasets. To deploy the state-of-the-art QA models to real-world applications, we need to construct high-quality datasets with large volumes of QA pairs to train them; however, this will be costly, requiring a massive amount of human efforts and time. Question generation (QG) , or Question-Answer pair generation (QAG) , is a popular approach to overcome this data scarcity challenge. Some of the recent works resort to semi-supervised learning, by leveraging large amount of unlabeled text (e.g. Wikipedia) to generate synthetic QA pairs with the help of QG systems (Tang et al., 2017; Yang et al., 2017; Tang et al., 2018; Sachan and Xing, 2018). However, existing QG systems have overlooked an important point that generating QA pairs from a context consisting of unstructured texts, is essentially a one-to-many problem. Sequence-to-sequence models are known to generate generic sequences (Zhao et al., 2017a) without much variety, as they are trained with maximum likelihood estimation. This is highly suboptimal for QAG since the contexts given to the model often contain richer information that could be exploited to generate multiple QA pairs. To tackle the above issue, we propose a novel probabilistic deep generative model for QA pair generation. Specifically, our model is a hierarchical conditional variational autoencoder (HCVAE) with two separate latent spaces for question and answer conditioned on the context, where the answer latent space is additionally conditioned on the question latent space. During generation, this hierarchical conditional VAE first generates an answer given a context, and then generates a question given both the answer and the context, by sampling from both latent spaces. This probabilistic approach allows the model to generate diverse QA pairs focusing on different parts of a context at each time. Another crucial challenge of the QG task is to ensure the consistency between a question and its corresponding answer, since they should be semantically dependent on each other such that the question is answerable from the given answer and the context. In this paper, we tackle this consistency issue by maximizing the mutual information (Bel-ghazi et al., 2018; Hjelm et al., 2019; Yeh and Chen, 2019) between the generated QA pairs. We empirically validate that the proposed mutual information maximization significantly improves the QA-pair consistency. Combining both the hierarchical CVAE and the InfoMax regularizer together, we propose a novel probabilistic generative QAG model which we refer to as Info rmation Maximizing H ierarchical C onditional V ariational A uto E ncoder ( Info-HCVAE ). Our Info-HCVAE generates diverse and consistent QA pairs even from a very short context (see Table 1). But how should we quantitatively measure the quality of the generated QA pairs? Popular evaluation metrics (e.g. BLEU (Papineni et al., 2002), ROUGE (Lin and Hovy, 2002), METEOR (Baner-jee and Lavie, 2005)) for text generation only tell how similar the generated QA pairs are to Ground-Truth (GT) QA pairs, and are not directly correlated with their actual quality (Nema and Khapra, 2018; Zhang and Bansal, 2019). Therefore, we use the QA -based E valuation ( QAE ) metric proposed by Zhang and Bansal (2019), which measures how well the generated QA pairs match the distribution of GT QA pairs. Yet, in a semi-supervised learning setting where we already have GT labels, we need novel QA pairs that are different from GT QA pairs for the additional QA pairs to be truly effective. Thus, we propose a novel metric, R everse QAE ( R-QAE ), which is low if the generated QA pairs are novel and diverse. We experimentally validate our QAG model on SQuAD v1.1 (Rajpurkar et al., 2016), Natural Questions (Kwiatkowski et al., 2019), and TriviaQA (Joshi et al., 2017) datasets, with both QAE and R-QAE using BERT-base (Devlin et al., 2019) as the QA model. Our QAG model obtains high QAE and low R-QAE, largely outperforming state-of-the-art baselines using a significantly smaller number of contexts. Further experimental results for semi-supervised QA on the three datasets using the SQuAD as the labeled dataset show that our model achieves significant improvements over the state-of-the-art baseline (+2.12 on SQuAD, +5.67 on NQ, and +1.18 on Trivia QA in EM). Our contribution is threefold:  We propose a novel hierarchical variational framework for generating diverse QA pairs from a single context, which is, to our knowledge, the first probabilistic generative model for question-answer pair generation (QAG). We propose an InfoMax regularizer which effectively enforces the consistency between the generated QA pairs, by maximizing their mutual information. This is a novel approach in resolving consistency between QA pairs for QAG. We evaluate our framework on several benchmark datasets by either training a new model entirely using generated QA pairs (QA-based evaluation), or use both ground-truth and generated QA pairs (semi-supervised QA). Our model achieves impressive performances on both tasks, largely outperforming existing QAG baselines. Question and Question-Answer Pair Generation Early works on Question Generation (QG) mostly resort to rule-based approaches (Heilman and Smith, 2010; Lindberg et al., 2013; Labutov et al., 2015). However, recently, encoder-decoder based neural architectures (Du et al., 2017; Zhou et al., 2017) have gained popularity as they outperform rule-based methods. Some of them use paragraph-level information (Du and Cardie, 2018; Song et al., 2018; Liu et al., 2019; Zhao et al., 2018; Kim et al., 2019; Sun et al., 2018) as additional information. Reinforcement learning is a popular approach to train the neural QG models, where the reward is defined as the evaluation metrics (Song et al., 2017; Kumar et al., 2018), or the QA ac-curacy/likelihood (Yuan et al., 2017; Hosking and Riedel, 2019; Zhang and Bansal, 2019). State-of-the-art QG models (Alberti et al., 2019; Dong et al., 2019; Chan and Fan, 2019) use pre-trained language models. Question-Answer Pair Generation (QAG) from contexts, which is our main target, is a relatively less explored topic tackled by only a few recent works (Du and Cardie, 2018; Alberti et al., 2019; Dong et al., 2019). To the best of our knowledge, we are the first to propose a probabilistic generative model for end-to-end QAG; Yao et al. (2018) use VAE for QG, but they do not tackle QAG. Moreover, we effectively resolve the QA-pair consistency issue by maximizing their mutual information with an InfoMax regularizer (Belghazi et al., 2018; Hjelm et al., 2019; Yeh and Chen, 2019), which is another contribution of our work. Semi-supervised QA with QG With the help of QG models, it is possible to train the QA models in a semi-supervised learning manner to obtain improved performance. Tang et al. (2017) apply dual learning to jointly train QA and QG on unlabeled dataset. Yang et al. (2017) and Tang et al. (2018) train QG and QA in a GAN framework (Goodfel-low et al., 2014). Sachan and Xing (2018) propose a curriculum learning to supervise the QG model to gradually generate difficult questions for the QA model. Dhingra et al. (2018) introduce a cloze-style QAG method to pretrain a QA model. Zhang and Bansal (2019) propose to filter out low-quality synthetic questions by the answer likelihood. While we focus on the answerable setting in this paper, few recent works tackle the unanswerable settings. Zhu et al. (2019) use neural networks to edit answerable questions into unanswerable ones, and perform semi-supervised QA. Alberti et al. (2019) and Dong et al. (2019) convert generated questions into unanswerable ones using heuristics, and filter or replace corresponding answers based on EM or F1. Variational Autoencoders Variational autoencoders (VAEs) (Kingma and Welling, 2014) are probabilistic generative models used in a variety of natural language understanding tasks, including language modeling (Bowman et al., 2016), dialogue generation (Serban et al., 2017; Zhao et al., 2017b; Park et al., 2018; Du et al., 2018; Qiu et al., 2019), and machine translation (Zhang et al., 2016; Su et al., 2018; Deng et al., 2018). In this work, we propose a novel hierarchical conditional VAE framework with an InfoMax regularization for generating a pair of samples with high consistency. Our goal is to generate diverse and consistent QA pairs to tackle the data scarcity challenge in the extractive QA task. Formally, given a context c which contains M tokens, c = ( c 1 , . . . , c M ) , we want to generate QA pairs ( x , y ) where x = ( x 1 , . . . , x N ) is the question containing N tokens and y = ( y 1 , . . . , y L ) is its corresponding answer containing L tokens. We aim to tackle the QAG task by learning the conditional joint distribution of the question and answer given the context, p ( x , y | c ) , from which we can sample the QA pairs: ( x , y )  p ( x , y | c ) We estimate p ( x , y | c ) with a probabilistic deep generative model, which we describe next. We propose to approximate the unknown conditional joint distribution p ( x , y | c ) , with a variational autoencoder (VAE) framework (Kingma and Welling, 2014). However, instead of directly learning a common latent space for both question and answer, we model p ( x , y | c ) in a hierarchical conditional VAE framework with a separate latent space for question and answer as follows: p  ( x , y | c ) = (cid:90) z x (cid:88) z y p  ( x | z x , y , c ) p  ( y | z x , z y , c )  p  ( z y | z x , c ) p  ( z x | c ) d z x where z x and z y are latent variables for question and answer respectively, and the p  ( z x | c ) and p  ( z y | z x , c ) are their conditional priors following an isotropic Gaussian distribution and a categorical distribution (Figure 1-(a)). We decompose the latent space of question and answer, since the answer is always a finite span of context c , which can be modeled well by a categorical distribution, while a continuous latent space is a more appropriate choice for question since there could be unlimited valid questions from a single context. Moreover, we design the bi-directional dependency flow of joint distribution for QA. By leveraging hierarchical structure, we enforce the answer latent variables Figure 1 : The conceptual illustration of the proposed HCVAE model encoding and decoding question and its corresponding answer jointly. The dashed line refers to the generative process of HCVAE. Figure 2 : The directed graphical model for HCVAE. The gray and white nodes denote observed and latent variables. to be dependent on the question latent variables in p  ( z y | z x , c ) and achieve the reverse dependency by sampling question x  p  ( x | z x , y , c ) . We then use a variational posterior q  (  ) to maximize the Evidence Lower Bound (ELBO) as follows (The complete derivation is provided in Appendix A): log p  ( x , y | c )  E z x  q  ( z x | x , c ) [log p  ( x | z x , y , c )] + E z y  q  ( z y | z x , y , c ) [log p  ( y | z y , c )]  DKL [ q  ( z y | z x , y , c ) || p  ( z y | z x , c )]  DKL [ q  ( z x | x , c ) || p  ( z x | c )] =: LHCVAE where  ,  , and  are the parameters of the generation, posterior, and prior network, respectively. We refer to this model as a Hierarchical Conditional Variational Autoencoder (HCVAE) framework. Figure 2 shows the directed graphical model of our HCVAE. The generative process is as follows: 1. Sample question L.V.: z x  p  ( z x | c ) 2. Sample answer L.V.: z y  p  ( z y | z x , c ) 3. Generate an answer: y  p  ( y | z y , c ) 4. Generate a question: x  p  ( x | z x , y , c ) Embedding We use the pre-trained word embedding network from BERT (Devlin et al., 2019) for posterior and prior networks, whereas the whole BERT is used as a contextualized word embedding model for the generative networks. For the answer encoding, we use a binary token type id of BERT. Specifically, we encode all context tokens as 0s, except for the tokens which are part of answer span (highlighted words of context in Figure 1-(a) or -(c)), which we encode as 1s. We then feed the sequence of the word token ids, token type ids, and position ids into the embedding layer to encode the answer-aware context. We fix all the embedding layers in HCVAE during training. Prior Networks We use two different conditional prior networks p  ( z x | c ) , p  ( z y | z x , c ) to model context-dependent priors (the dashed lines in Figure 1-(a)). To obtain the parameters of isotropic Gaussian N (  ,  2 I ) for p  ( z x | c ) , we use a bidirectional LSTM (Bi-LSTM) to encode the word embeddings of the context into the hidden representations, and then feed them into a Multi-Layer Perceptron (MLP). We model p  ( z y | z x , c ) following a categorical distribution Cat (  ) , by computing the parameter  from z x and the hidden representation of the context using another MLP. Posterior Networks We use two conditional posterior networks q  ( z x | x , c ) , q  ( z y | z x , y , c ) to approximate true posterior distributions of latent variables for both question x and answer y . We use two Bi-LSTM encoders to output the hidden representations of question and context given their word embeddings. Then, we feed the two hidden representations into MLP to obtain the parameters of Gaussian distribution,  (cid:48) and  (cid:48) (upper right corner in Figure 1-(a)). We use the reparameterization trick (Kingma and Welling, 2014) to train the model with backpropagation since the stochastic sampling process z x  q  ( z x | x , c ) is nondifferentiable. We use another Bi-LSTM to encode the word embedding of answer-aware context into the hidden representation. Then, we feed the hidden representation and z x into MLP to compute the parameters  (cid:48) of categorical distribution (lower right corner in Figure 1-(a)). We use the categorical reparameterization trick with gumbel-softmax (Maddison et al., 2017; Jang et al., 2017) to enable backpropagation through sampled discrete latent variables. Answer Generation Networks Since we consider extractive QA, we can factorize p  ( y | z y , c ) into p  ( y s | z y , c ) and p  ( y e | z y , c ) , where y s and y e are the start and the end position of an answer span (highlighted words in Figure 1-(b)), respectively. To obtain MLE estimators for both, we first encode the context c into the contextualized word embedding of E c = { e c 1 , . . . , e c M } with the pre-trained BERT. We compute the final hidden representation of context and the latent variable z y with a heuristic matching layer (Mou et al., 2016) and a Bi-LSTM: f i = [ e c i ; z y ; | e c i  z y | ; e c i (cid:12) z y ]  h i =  LSTM ([ f i ,  h i  1 ])  h i =  LSTM ([ f i ,  h i +1 ]) H = [  h i ;  h i ] Mi =1 where z y is linearly transformed, and H  R d y  M is the final hidden representation. Then, we feed H into two separate linear layers to predict y s and y e . Question Generation Networks We design the encoder-decoder architecture for our QG network by mainly adopting from our baselines (Zhao et al., 2018; Zhang and Bansal, 2019). For encoding, we use pre-trained BERT to encode the answer-specific context into the contextualized word embedding, and then use a two-layer Bi-LSTM to encode it into the hidden representation (in Figure 1-(c)). We apply a gated self-attention mechanism (Wang et al., 2017) to the hidden representation to better capture long-term dependencies within the context, to obtain a new hidden representation H  R d x  M . The decoder is a two-layered LSTM which receives the latent variable z x as an initial state. It uses an attention mechanism (Luong et al., 2015) to dynamically aggregate  H at each decoding step into a context vector of s j , using the j -th decoder hidden representation d j  R d x (in Figure 1-(c)). Then, we feed d j and s j into MLP with maxout activation (Goodfellow et al., 2013) to compute the final hidden representation d j as follows: d 0 = z x , d j = LSTM ([ e x j  1 , d j  1 ]) r j = H TW a d j , a j = softmax ( r j ) , s j = Ha j  d j = MLP ([ d j ; s j ]) where z x is linearly transformed, and e x j is the j -th question word embedding. p ( x j | x <j , z x , y , c ) = softmax ( W e d j ) . We initialize the weight matrix W e as the pretrained word embedding matrix and fix it during training. Further, we use the copy mechanism (Zhao et al., 2018), so that the model can directly copy tokens from the context. We also greedily decode questions to ensure that all stochasticity comes from the sampling of the latent variables. One of the most important challenges of the QAG task is enforcing consistency between the generated question and its corresponding answer. They should be semantically consistent, such that it is possible to predict the answer given the question and the context. However, neural QG or QAG models often generate questions irrelevant to the context and the answer (Zhang and Bansal, 2019) due to the lack of the mechanism enforcing this consistency. We tackle this issue by maximizing the mutual information (MI) of a generated QA pair, assuming that an answerable QA pair will have high MI. Since an exact computation of MI is intractable, we use a neural approximation. While there exist many different approximations (Belg-hazi et al., 2018; Hjelm et al., 2019), we use the estimation proposed by Yeh and Chen (2019) based on Jensen-Shannon Divergence: MI ( X ; Y )  E x , y  P [log g ( x , y )] + 1 2 E  x , y  N [log(1  g ( x , y ))] + 1 2 E x ,  y  N [log(1  g ( x ,  y ))] =: L Info where EP and EN denote expectation over positive and negative examples. We generate negative examples by shuffling the QA pairs in the minibatch, such that a question is randomly associated with an answer. Intuitively, the function g (  ) acts like a binary classifier that discriminates whether QA pair is from joint distribution or not. We empirically find that the following g (  ) effectively achieves our goal of consistent QAG: g ( x , y ) = sigmoid ( x T Wy ) where x = 1 N (cid:80) i d i and y = 1 L (cid:80) j h j are summarized representations of question and answer, respectively. Combined with the ELBO, the final objective of our Info-HCVAE is as follows: max  LHCVAE +  L Info where  includes all the parameters of , ,  and W , and  controls the effect of MI maximization. Stanford Question Answering Dataset v1.1 (SQuAD) (Rajpurkar et al., 2016). This is a reading comprehension dataset consisting of questions obtained from crowdsourcing on a set of Wikipedia articles, where the answer to every question is a segment of text or a span from the corresponding reading passage. We use the same split used in Zhang and Bansal (2019) for the fair comparison. Natural Questions (NQ) (Kwiatkowski et al., 2019). This dataset contains realistic questions from actual user queries to a search engine, using Wikipedia articles as context. We adapt the dataset provided from MRQA shared task (Fisch et al., 2019) and convert it into the extractive QA format. We split the original validation set in half, to use as validation and test for our experiments. TriviaQA (Joshi et al., 2017). This is a reading comprehension dataset containing question-answer-evidence triples. The QA pairs and the evidence (contexts) documents are authored and uploaded by Trivia enthusiasts. Again, we only choose QA pairs of which answers are span of contexts. HarvestingQA 2 This dataset contains top-ranking 10K Wikipedia articles and 1M synthetic QA pairs generated from them, by the answer span extraction and QG system proposed in (Du and Cardie, 2018). We use this dataset for semi-supervised learning. Implementation Details In all experiments, we use BERT-base ( d = 768 ) (Devlin et al., 2019) as the QA model, setting most of the hyperparameters as described in the original paper. For both HCVAE and Info-HCVAE, we set the hidden dimensionality of the Bi-LSTM to 300 for posterior, prior, and answer generation networks, and use the dimensionality of 450 and 900 for the encoder and the decoder of the question generation network. We set the dimensionality of z x as 50 , and define z y to be set of 2 https://github.com/xinyadu/ harvestingQA 10-way categorical variables z y = { z 1 , . . . , z 20 } . For training the QA model, we fine-tune the model for 2 epochs. We train both the QA model and Info-HCVAE with Adam optimizer (Kingma and Ba, 2015) with the batch size of 32 and the initial learning rate of 5  10  5 and 10  3 respectively. For semi-supervised learning, we first pre-train BERT on the synthetic data for 2 epochs and fine-tune it on the GT dataset for 2 epochs. To prevent posterior collapse , we multiply 0 . 1 to the KL divergence terms of question and answer (Higgins et al., 2017). For more details of the datasets and experimental setup, please see Appendix C. Baselines We experiment two variants of our model against several baselines: 1. Harvest-QG : An attention-based neural QG model with a neural answer extraction system (Du and Cardie, 2018). 2. Maxout-QG : A neural QG model based on maxout copy mechanism with a gated self-attetion (Zhao et al., 2018), which uses BERT as the word embedding as suggested by Zhang and Bansal (2019). 3. Semantic-QG : A neural QG model based on Maxout-QG with semantic-enhanced reinforcement learning (Zhang and Bansal, 2019). 4. HCVAE : Our HCVAE model without the InfoMax regularizer. 5. Info-HCVAE : Our full model with the InfoMax regularizer. For the baselines, we use the same answer spans extracted by the answer extraction system (Du and Cardie, 2018). QAE and R-QAE One of crucial challenges with generative models is a lack of a good quantitative evaluation metric. We adopt QA -based E valuation (QAE) metric proposed by Zhang and Bansal (2019) to measure the quality of QA pair. QAE is obtained by first training the QA model on the synthetic data, and then evaluating the QA model with human annotated test data. However, QAE only measures how well the distribution of synthetic QA pairs matches the distribution of GT QA pairs, and does not consider the diversity of QA pairs. Thus, we propose R everse QA -based E valuation ( R-QAE ), which is the accuracy of the QA model trained on the human-annotated QA pairs, evaluated on the generated QA pairs. If the synthetic Method QAE (  ) R-QAE (  ) SQuAD (EM/F1) Harvesting-QG 55.11/66.40 64.77/78.85 Maxout-QG 56.08/67.50 62.49/78.24 Semantic-QG 60.49/71.81 74.23/88.54 HCVAE 69.46/80.79 37.57 /61.24 Info-HCVAE 71.18/81.51 38.80/ 60.73 Natural Questions (EM/F1) Harvesting-QG 27.91/41.23 49.89/70.01 Maxout-QG 30.98/44.96 49.96/70.03 Semantic-QG 30.59/45.29 58.42/79.23 HCVAE 31.45/46.77 32.78/55.12 Info-HCVAE 37.18/51.46 29.39/53.04 TriviaQA (EM/F1) Harvesting-QG 21.32/30.21 29.75/47.73 Maxout-QG 24.58/34.32 31.56/49.92 Semantic-QG 27.54/38.25 37.45/58.15 HCVAE 30.20/40.88 34.41/48.16 Info-HCVAE 35.45/44.11 21.65/37.65 Table 2 : QAE and R-QAE results on three datasets. All results are the performances on our test set. Table 3 : The results of mutual information estimation. The results are based on QA pairs generated from H  10%. data covers larger distribution than the human annotated training data, R-QAE will be lower. However, note that having a low R-QAE is only meaningful when the QAE is high enough since trivially invalid questions may also yield low R-QAE. Results We compare HCVAE and Info-HCVAE with the baseline models on SQuAD, NQ, and TriviaQA. We use 10% of Wikipedia paragraphs from HarvestingQA (Du and Cardie, 2018) for evaluation. Table 2 shows that both HCVAE and Info-HCVAE significantly outperforms all baselines by large margin in QAE on all three datasets, while obtaining significantly lower R-QAE, which shows that our model generated both high-quality and diverse QA pairs from the given context. Moreover, Info-HCVAE largely outperforms HCVAE, which demonstrates the effectiveness of our InfoMax regularizer for enforcing QA-pair consistency. Figure 3 shows the accuracy as a function of number of QA pairs. Our Info-HCVAE outperform all baselines by large margins using orders of magnitude smaller number of QA pairs. For example, Info-HCVAE achieves 61 . 38 points using 12K QA pairs, outperforming Semantic-QG that use 10 times larger number of QA pairs. We also report 10 4 10 5 10 6 50 60 70 # of QA pairs (log-scaled) QAb a s e d E v a l u a ti on ( EM ) Harvest-QG Maxout-QG Semantic-QG Info-HCVAE Figure 3: QAE vs. # of QA pairs (log-scaled) on SQuAD. Table 4 : QAE and R-QAE results of the ablation study SQuAD dataset. All the results are the performances on our test set. the score of x T Wy as an approximate estimation of mutual information (MI) between QA pairs generated by each method in Table 3; our Info-HCVAE yields the largest value of MI estimation. Ablation Study We further perform an ablation study to see the effect of each model component. We start with the model without any latent variables, which is essentially a deterministic Seq2Seq model (denoted as Baseline in Table 4). Then, we add in the question latent variable (+Q-latent) and then the answer latent variable (+A-latent), to see the effect of probabilistic latent variable modeling and hierarchical modeling respectively. The results in Table 4 shows that both are essential for improving both the quality (QAE) and diversity (R-QAE) of the generated QA pairs. Finally, adding in the InfoMax regularization (+InfoMax) further improves the performance by enhancing the consistency of the generated QA pairs. Human Evaluation As a qualitative analysis, we first conduct a pairwise human evaluation of the QA pairs generated by our Info-HCVAE and Maxout-QG on 100 randomly selected paragraphs. Specifically, 20 human judges performed blind quality assessment of two sets of QA pairs that are presented in a random order, each of which contained two to five QA pairs. Each set of QA pairs is evalu-Method Diversity Consistency Overall Baseline 26% 34% 30% Ours 47% 50% 52% Tie 27% 16% 18% Table 5 : The results of human judgement in terms of diversity, consistency, and overall quality on the generated QA pairs. Table 6 : Examples of one-to-many mapping of our Info-HCVAE. The answer is highlighted by pink. GT denotes the ground-truth question. Odenotes questions generated by Info-HCVAE. ated in terms of the overall quality, diversity, and consistency between the generated QA pairs and the context. The results in Table 5 show that the QA pairs generated by our Info-HCVAE is evaluated to be more diverse and consistent, compared to ones generated by the baseline models. One-to-Many QG To show that our Info-HCVAE can effectively tackle one-to-many mapping problem for question generation, we qualitatively analyze the generated questions for given a context and an answer from the SQuAD validation set. Specifically, we sample the question latent variables multiple times using the question prior network p  ( z x | c ) , and then feed them to question generation networks p  ( x | z x , y , c ) with the answer. The example in Table 6 shows that our Info-HCVAE generates diverse and semantically consistent questions given an answer. We provide more qualitative examples in Appendix D. Latent Space Interpolation To examine if Info-HCVAE learns meaningful latent space of QA pairs, we qualitatively analyze the QA pairs generated by interpolating between two latent codes of it on SQuAD training set. We first encode z x from two QA pairs using posterior networks of q  ( z x | x , c ) , and then sample z y from interpolated values of z x using prior networks p  ( z y | z x , c ) to generate corresponding QA pairs. Table 7 shows that the semantic of the QA pairs generated smoothly transit from one latent to another with high diversity and consistency. We provide more qualitative examples Paragraph ... Table 7 : QA pairs generated by interpolating between two latent codes encoded by our posterior networks. Ori1 and Ori2 are from training set of SQuAD. We now validate our model in a semi-supervised setting, where the model uses both the ground truth labels and the generated labels to solve the QA task, to see whether the generated QA pairs help improve the performance of a QA model in a conventional setting. Since such synthetic datasets consisting of generated QA pairs may inevitably contain some noise (Zhang and Bansal, 2019; Dong et al., 2019; Alberti et al., 2019), we further refine the QA pairs by using the heuristic suggested by Dong et al. (2019), to replace the generated answers whose F1 score to the prediction of the QA model trained on the human annotated data is lower than a set threshold. We select the threshold of 40 . 0 for the QA pair refinement model via cross-validation on the SQuAD dataset, and used it for the experiments. Please see Appendix C for more details. SQuAD We first perform semi-supervised QA experiments on SQuAD using the synthetic QA pairs generated by our model. For the contexts, we use both the paragraphs in the original SQuAD (S) dataset, and the new paragraphs in the HarvestingQA dataset (H). Using Info-HCVAE, we generate 10 different QA pairs by sampling from the latent spaces (denoted as S  10). For the baseline, we use Semantic-QG (Zhang and Bansal, 2019) with the beam search size of 10 to obtain the same number of QA pairs. We also generate new QA pairs Data EM F1 SQuAD 80.25 88.23 Semantic-QG (baseline) +S  10 81.20 (+0.95) 88.36 (+0.13) +H  100% 81.03 (+0.78) 88.79 (+0.56) +S  10 + H  100% 81.44 (+1.19) 88.72 (+0.49) Info-HCVAE (ours) +S  10 82.09 (+1.84) 89.11 (+0.88) +H  10% 81.37 (+1.12) 88.85 (+0.62) +H  20% 81.68 (+1.43) 89.06 (+0.93) +H  30% 81.76 (+1.51) 89.12 (+0.89) +H  50% 82.17 (+1.92) 89.38 (+1.15) +H  100% 82.37 (+2.12) 89.63 (+1.40) +S  10 + H  100% 82.19 (+1.94) 89.84 (+1.59) Table 8 : The results of semi-supervised QA experiments on SQuAD. All the results are the performances on our test set. using different portions of paragraphs provided in HarvestingQA (denoted as H  10%-H  100%), by sampling one latent variable per context. Table 8 shows that our framework improves the accuracy of the BERT-base model by 2.12 (EM) and 1.59 (F1) points, significantly outperforming Semantic-QG. NQ and TriviaQA Our model is most useful when we do not have any labeled data for a target dataset. To show how well our QAG model performs in such a setting, we train the QA model using only the QA pairs generated by our model trained on SQuAD and test it on the target datasets (NQ and TriviaQA). We generate multiple QA pairs from each context of the target dataset, sampling from the latent space one to ten times (denoted by N  1-10 or T  1-10 in Table 9). Then, we fine-tune the QA model pretrained on the SQuAD dataset with the generated QA pairs from the two datasets. Table 9 shows that as we augment training data with larger number of synthetic QA pairs, the performance of the QA model significantly increases, significantly outperforming the QA model trained on SQuAD only. Yet, models trained with our QAG still largely underperform models trained with human labels, due to the distributional discrepancy between the source and the target dataset. We proposed a novel probabilistic generative framework for generating diverse and consistent question-answer (QA) pairs from given texts. Specifically, our model learns the joint distribution of question and answer given context with a hierarchically conditional variational autoencoder, while enforcing consistency between generated QA pairs by maximizing their mutual information with a novel In-Data EM F1 Natural Questions SQuAD 42.77 57.29 +N  1 46.70 (+3.94) 61.08 (+3.79) +N  2 46.95 (+4.19) 61.34 (+4.05) +N  3 47.73 (+4.96) 61.98 (+4.69) +N  5 48.19 (+5.42) 62.21 (+4.92) +N  10 48.44 (+5.67) 62.69 (+5.40) NQ 61.65 73.91 TriviaQA SQuAD 48.96 57.98 +T  1 49.65 (+0.69) 59.13 (+1.21) +T  2 50.01 (+1.05) 59.08 (+1.10) +T  3 49.71 (+0.75) 59.49 (+1.51) +T  5 50.14 (+1.18) 59.21 (+1.23) +T  10 49.65 (+0.69) 59.20 (+1.22) Trivia 64.55 70.42 Table 9 : The result of semi-supervised QA experiments on Natural Questions and TriviaQA dataset. All results are the performance on our test set. foMax regularizer. To our knowledge, ours is the first successful probabilistic QAG model. We evaluated the QAG performance of our model by the accuracy of the BERT-base QA model trained using the generated questions on multiple datasets, on which it largely outperformed the state-of-the-art QAG baseline (+6.59-10.69 in EM), even with a smaller number of QA pairs. We further validated our model for semi-supervised QA, where it improved the performance of the BERT-base QA model on the SQuAD by 2.12 in EM, significantly outperforming the state-of-the-art model. As future work, we plan to extend our QAG model to a meta-learning framework, for generalization over diverse datasets. This work was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF-2018R1A5A1059921), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea govern-ment(MSIT) (No.2019-0-01410, Research Development of Question Generation for Deep Learning based Semantic Search Domain Extension, No.2016-0-00563, Research on Adaptive Machine Learning Technology Development for Intelligent Autonomous Digital Companion, No.20190-00075, and Artificial Intelligence Graduate School Program (KAIST)).
abstain abstain objective method result abstain other other abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain objective abstain abstain abstain abstain method objective objective result method other abstain objective objective method result result objective objective objective objective result other other other other other method abstain abstain other other other other other other method other other other objective method method method method method abstain method method method abstain abstain abstain abstain method method method method method method abstain method method method method abstain method method method method method abstain method method method abstain method method abstain method method method method method abstain method abstain abstain abstain method abstain method method method abstain abstain abstain method method other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method other method method method method abstain method abstain abstain method method method abstain abstain abstain abstain abstain other result abstain abstain abstain abstain result abstain result abstain abstain result abstain abstain abstain abstain result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain result other result abstain result abstain result abstain abstain abstain abstain objective other result abstain objective result result objective other