Weakly supervised question answering usually has only the final answers as supervision signals while the correct solutions to derive the answers are not provided. This setting gives rise to the spurious solution problem : there may exist many spurious solutions that coincidentally derive the correct answer, but training on such solutions can hurt model performance (e.g., producing wrong solutions or answers). For example, for discrete reasoning tasks as on DROP, there may exist many equations to derive a numeric answer, and typically only one of them is correct. Previous learning methods mostly filter out spurious solutions with heuristics or using model confidence, but do not explicitly exploit the semantic correlations between a question and its solution. In this paper, to alleviate the spurious solution problem, we propose to explicitly exploit such semantic correlations by maximizing the mutual information between question-answer pairs and predicted solutions. Extensive experiments on four question answering datasets show that our method significantly outperforms previous learning methods in terms of task performance and is more effective in training models to produce correct solutions. Weakly supervised question answering is a common setting of question answering (QA) where only final answers are provided as supervision signals while the correct solutions to derive them are not. This setting simplifies data collection, but exposes model learning to the spurious solution problem : there may exist many spurious ways to derive the correct answer, and training a model with spurious solutions can hurt model performance (e.g., misleading the model to produce unreasonable solutions or wrong answers). As shown in Fig 1,  *Corresponding author: Minlie Huang. for multi-mention reading comprehension, many mentions of an answer in the document(s) are irrelevant to the question; for discrete reasoning tasks or text2SQL tasks, an answer can be produced by the equations or SQL queries that do not correctly match the question in logic. Some previous works heuristically selected one possible solution per question for training, e.g., the first answer span in the document (Joshi et al., 2017; Tay et al., 2018; Talmor and Berant, 2019); some treated all possible solutions equally and maximized the sum of their likelihood (maximum marginal likelihood, or MML) (Swayamdipta et al., 2018; Clark and Gardner, 2018; Lee et al., 2019); many others selected solutions according to model confidence (Liang et al., 2018; Min et al., 2019), i.e., the likelihood of the solutions being derived by the model. A drawback of these methods is that they do not explicitly consider the mutual semantic correlations between a question and its solution when selecting solutions for training. Intuitively speaking, a question often contains vital clues about how to derive the answer, and a wrong solution together with its context often fails to align well with the question. Take the discrete reasoning case in Fig 1 as an example. To answer the question, we need to know the start year of the Battle of Powder River , which is answered by the first 1876 ; the second 1876 is irrelevant as it is the year of an event that happened during the battle. To exploit the semantic correlations between a question and its solution, we propose to maximize the mutual information between question-answer pairs and model-predicted solutions. As demonstrated by Min et al. (2019), for many QA tasks, it is feasible to precompute a modestly-sized, task-specific set of possible solutions containing the correct one. Therefore, we focus on handling the spurious solution problem under this circumstance. Specifically, we pair a task-specific model with a question reconstructor and repeat the following training cycle (Fig 2): (1) sample a solution from the solution set according to model confidence, train the question reconstructor to reconstruct the question from that solution, and then (2) train the task-specific model on the most likely solution according to the question reconstructor. During training, the question reconstructor guides the task-specific model to predict those solutions consistent with the questions. For the question reconstructor, we devise an effective and unified way to encode solutions in different tasks, so that solutions with subtle differences (e.g., different spans with the same surface form) can be easily discriminated. Our contributions are as follows: (1) We propose a mutual information maximization approach for the spurious solution problem in weakly supervised QA, which exploits the semantic correlations between a question and its solution; (2) We conducted extensive experiments on four QA datasets. Our approach significantly outperforms strong baselines in terms of task performance and is more effective in training models to produce correct solutions. Question answering has raised prevalent attention and has achieved great progress these years. A lot of challenging datasets have been constructed to advance models' reasoning abilities, such as (1) reading comprehension datasets with extractive answer spans (Joshi et al., 2017; Dhingra et al., 2017), with free-form answers (Kocisky et al., 2018), for multi-hop reasoning (Yang et al., 2018), or for discrete reasoning over paragraphs (Dua et al., 2019), and (2) datasets for semantic parsing (Pasupat and Liang, 2015; Zhong et al., 2017; Yu et al., 2018). Under the weakly supervised setting, the specific solutions to derive the final answers (e.g., the correct location of an answer text, or the correct logic executing an answer) are not provided. This setting is worth exploration as it simplifies annotation and makes it easier to collect large-scale corpora. However, this setting introduces the spurious solution problem, and thus complicates model learning. Most existing approaches for this learning challenge include heuristically selecting one possible solution per question for training (Joshi et al., 2017; Tay et al., 2018; Talmor and Berant, 2019), training on all possible solutions with MML (Swayamdipta et al., 2018; Clark and Gardner, 2018; Lee et al., 2019; Wang et al., 2019), reinforcement learning (Liang et al., 2017, 2018), and hard EM (Min et al., 2019; Chen et al., 2020). All these approaches either use heuristics to select possibly reasonable solutions, rely on model architectures to bias towards correct solutions, or use model confidence to filter out spurious solutions in a soft or hard way. They do not explicitly exploit the semantic correlations between a question and its solution. Most relevantly, Cheng and Lapata (2018) focused on text2SQL tasks; they modeled SQL queries as the latent variables for question generation, and maximized the evidence lower bound of log likelihood of questions. A few works treated solution prediction and question generation as dual tasks and introduced dual learning losses to regularize learning under the fully-supervised or the semi-supervised setting (Tang et al., 2017; Cao et al., 2019; Ye et al., 2019). In dual learning, a model generates intermediate outputs (e.g., the task-specific model predicts solutions from a question) while the dual model gives feedback signals (e.g., the question reconstructor computes the likelihood of the question conditioned on predicted solutions). This method is featured in three aspects. First, both models need training on fully-annotated data so that they can produce reasonable intermediate outputs. Second, the intermediate outputs can introduce noise during learning as they are sampled from models but not restricted to solutions with correct answer or valid questions. Third, this method typically updates both models with reinforcement learning while the rewards provided by a dual model can be unstable or of high variance. By contrast, we focus on the spurious solution problem under the weakly supervised setting and propose a mutual information maximization approach. Solutions used for training are restricted to those with correct answer. What's more, though a task-specific model and a question reconstructor interact with each other, they do not use the likelihood from each other as rewards, which can stabilize learning. For a QA task, each instance is a tuple (cid:104) d , q , a (cid:105) , where q denotes a question, a is the answer, and d is reference information such as documents for reading comprehension, or table headers for semantic parsing. A solution z is a task-specific derivation of the answer, e.g., a particular span in a document, an equation, or a SQL query (as shown in Fig 1). Let f (  ) be the task-specific function that maps a solution to its execution result, e.g., by returning a particular span, solving an equation, or executing a SQL query. Our goal is to train a task-specific model P  ( z | d, q ) that takes (cid:104) d , q (cid:105) as input and predicts a solution z satisfying f ( z ) = a . Under the weakly supervised setting, only the answer a is provided for training while the ground-truth solution  z is not. We denote the set of possible solutions as Z = { z | f ( z ) = a } . In cases where the search space of solution is large, we can usually approximate Z so that it contains the ground-truth solution  z with a high probability (Min et al., 2019; Wang et al., 2019). Note that Z is task-specific, which will be instantiated in section 4. During training, we pair the task-specific model P  ( z | d, q ) with a question reconstructor P  ( q | d, z ) and maximize the mutual information between (cid:104) q , a (cid:105) and z . During test, given (cid:104) d , q (cid:105) , we use the task-specific model to predict a solution and return the execution result. Given an instance (cid:104) d , q , a (cid:105) , the solution set Z usually contains only one solution that best fits the instance while the rest are spurious. We propose to exploit the semantic correlations between a ques-A Case of Discrete Reasoning over Paragraphs Question q = How many years afterthe Battle of Powder Riverdid PowervilleMontana become the first establishment in the county? tion and its solution to alleviate the spurious solution problem via mutual information maximization. Our objective is to obtain the optimal task-specific model   that maximizes the following conditional mutual information:   = arg max  I  ( (cid:104) q, a (cid:105) ; z | d ) = arg max  H ( (cid:104) q, a (cid:105)| d )  H  ( (cid:104) q, a (cid:105)| d, z ) = arg max   H  ( (cid:104) q, a (cid:105)| d, z ) = arg max  EP ( d,q,a ) EP  ( z | d,q,a ) log P  ( q, a | d, z ) (1) where I  ( (cid:104) q, a (cid:105) ; z | d ) denotes conditional mutual information between (cid:104) q, a (cid:105) and z over P ( d, q, a ) P  ( z | d, q, a ) . H ( | ) is conditional entropy of random variable(s). P ( d, q, a ) is the probability of an instance from the training distribution. P  ( z | d, q, a ) is the posterior prediction probability of z (  Z ) which is the prediction probability P  ( z | d, q ) normalized over Z : P  ( z | d, q, a ) = (cid:40) P  ( z | d,q ) (cid:80) z (cid:48) ZP  ( z (cid:48) | d,q ) z  Z 0 z /  Z (2) Note that computing P  ( q, a | d, z ) is intractable. We therefore introduce a question reconstructor P  ( q | d, z ) and approximate P  ( q, a | d, z ) with I ( f ( z ) = a ) P  ( q | d, z ) where I (  ) denotes indicator function. Eq. 1 now becomes:   = arg max  L 1 + L 2 L 1 = EP ( d,q,a ) EP  ( z | d,q,a ) log P  ( q | d, z ) L 2 = EP ( d,q,a ) EP  ( z | d,q,a ) log P  ( q, a | d, z ) P  ( q | d, z ) (3) To optimize Eq. 3 is to repeat the following training cycle, which is analogous to the EM algorithm: 1. Minimize L 2 w.r.t. the question reconstructor  to draw P  ( q | d, z ) close to P  ( q, a | d, z ) , by sampling a solution z (cid:48)  Z according to its posterior prediction probability P  ( z | d, q, a ) (see Eq. 2) and maximizing log P  ( q | d, z (cid:48) ) . Bart Encoder <s> a b <sol> op1 <span> op2 </s> Refers to Reference Infomation Solution 2. Maximize L 1 w.r.t. the task-specific model  . L 1 can be seen as a reinforcement learning objective with log P  ( q | d, z ) being the reward function. During training, the reward function is dynamically changing and may be of high variance. As we can compute the reward for all z  Z , we therefore adopt a greedy but more stable update method, i.e., to maximize log P  ( z (cid:48)(cid:48) | d, q ) where z (cid:48)(cid:48) = arg max z  Z log P  ( q | d, z ) is the best solution according to the question reconstructor. We illustrate the above training cycle in Fig 2. 3.3 Question Reconstructor The question reconstructor P  ( q | d, z ) takes reference information d and a solution z as input, and reconstructs the question q . We use BART base , a pre-trained Seq2Seq model, as the question reconstructor so that semantic correlations between questions and solutions can be better captured. A solution typically consists of task-specific operation token(s) (e.g., COUNT for discrete reasoning or semantic parsing), literal(s) (e.g., numeric constants for discrete reasoning or semantic pars-ing), or span(s) from a question or reference information (e.g., for most QA tasks). It is problematic to just feed the concatenation of d and the surface form of z to the BART encoder; otherwise, different spans with the same surface form can no longer be discriminated as their contextual semantics are lost. To effectively encode d and z , we devise a unified solution encoding as in Fig 3 which is applicable to solutions of various types. Specifically, we leave most of the surface form of z unchanged, except that we replace any span from reference information with a placeholder (cid:104) span (cid:105) . The representation of (cid:104) span (cid:105) is computed by forcing it to only attend to the contextual representation(s) of the referred span. To obtain disentangled and robust representations of reference information and a solution, we keep reference information and the solution (except for the token (cid:104) span (cid:105) ) from attending to each other. Intuitively speaking, semantics of reference information should not be affected by a solution, and the representations of a solution should largely determined by its internal logic. While our learning method and question reconstructor are task-agnostic, solutions are usually task-specific. Precomputing solution sets needs formal definitions of solutions which define the search space of solutions. A possible search method is to exhaustively enumerate all solutions that produce the correct answer. We will introduce the definitions of solutions for different tasks in section 4. Following Min et al. (2019), we conducted experiments on three QA tasks, namely multi-mention reading comprehension, discrete reasoning over paragraphs, and semantic parsing. This section introduces baselines, the definitions of solutions in different tasks, how the solution set can be precomputed, and our experimental results. Statistics of the datasets we used are presented in Table 1. For convenience, we denote reference information as d = [ d 1 , d 2 , ..., d | d | ] and denote a question as q = [ q 1 , q 2 , ..., q | q | ] where d i and q j are a token of d and q respectively. A span from reference information and a question span is represented as ( s, e ) d and ( s, e ) q respectively, where s and e are start and end index of the span respectively. First Only (Joshi et al., 2017) which trains a reading comprehension model by maximizing log P  ( z | d, q ) where z is the first answer span in d . MML (Min et al., 2019) which maximizes log (cid:80) z  ZP  ( z | d, q ) . HardEM (Min et al., 2019) which maximizes log max z  ZP  ( z | d, q ) . HardEM-thres (Chen et al., 2020): a variant of HardEM that optimizes only on confident solutions, i.e., to maximize max z  ZI ( P  ( z | d, q ) >  ) log P  ( z | d, q ) where  is an exponentially decaying threshold. is initialized such that a model is trained on no less than half of training data at the first epoch. We halve  after each epoch. VAE (Cheng and Lapata, 2018): a method that views a solution as the latent variable for question generation and adopts the training objective of Variational Auto-Encoder (VAE) (Kingma and Welling, 2014) to regularize the task-specific model. The overall training objective is given by:   ,   = arg max , L ( ,  ) L ( ,  ) = L mle (  ) +  L vae ( ,  ) = (cid:88) z  B log P  ( z | d, q ) + E P  ( z | d,q ) log P  ( q | d, z ) P  ( z | d, q ) where  denotes a task-specific model and  is our question reconstructor. L mle (  ) is the total log likelihood of the set of model-predicted solutions (denoted by B ) which derive the correct answer. L vae ( ,  ) is the evidence lower bound of the log likelihood of questions. is the coefficient of L vae ( ,  ) . This method needs pre-training both  and  before optimizing the overall objective L ( ,  ) . Notably, model  optimizes on L vae ( ,  ) via reinforcement learning. We tried stabilizing training by reducing the variance of rewards and setting a small  . Multi-mention reading comprehension is a natural feature of many QA tasks. Given a document d and a question q , a task-specific model is required to locate the answer text a which is usually mentioned many times in the document(s). A solution is defined as a document span. The solution set Z is computed by finding exact match of a : Z = { z = ( s, e ) d | [ d s , ..., d e ] = a } We experimented on two open domain QA datasets, i.e., Quasar-T (Dhingra et al., 2017) and WebQuestions (Berant et al., 2013). For Quasar-T, we retrieved 50 reference sentences from ClueWeb09 for each question; for WebQuestions, we used the 2016-12-21 dump of Wikipedia as the knowledge source and retrieved 50 reference paragraphs for each question using a Lucene index system. We used the same BERT base (Devlin et al., 2019) reading comprehension model and data preprocessing from (Min et al., 2019). Results: Our method outperforms all baselines on both datasets (Table 2). The improvements can be attributed to the effectiveness of solution encoding, as solutions for this task are typically different spans with the same surface form, e.g., in Qusart-T, all z  Z share the same surface form. Some reading comprehension tasks pose the challenge of comprehensive analysis of texts by requiring discrete reasoning (e.g., arithmetic calculation, sorting, and counting) (Dua et al., 2019). In this task, given a paragraph d and a question q , an answer a can be one of the four types: numeric value, a paragraph span or a question span, a sequence of paragraph spans, and a date from the paragraph. The definitions of z depend on answer types (Table 4). These solutions can be searched by following Chen et al. (2020). Note that some solutions involve numbers in d . We treated those numbers as spans while reconstructing q from z . analysis, we used the public development set as our test set, and split the public train set into 90%/10% for training and development. We used Neural Symbolic Reader (NeRd) (Chen et al., 2020) as the task-specific model. NeRd is a Seq2Seq model which encodes a question and a paragraph, and decodes a solution (e.g., count (paragraph span( s 1 , e 1 ), paragraph span( s 2 , e 2 )) where paragraph span( s i , e i ) means a paragraph span starting at s i and ending at e i ). We used the precomputed solution sets provided by Chen et al. (2020) 1 . Data preprocessing 1 Our implementation of NeRd has four major differences from that of (Chen et al., 2020). (1) Instead of choosing BERT large as encoder, we chose the discriminator of Electra base (Clark et al., 2020) which is of a smaller size. (2) We did not use moving averages of trained parameters. (3) We did not use the full public train set for training but used 10% of it for development. (4) For some questions, it is hard to guarantee that a precomputed solution set covers the ground-truth solution. For example, the question How many touchdowns did was also kept the same. Results: As shown in Table 3, our method significantly outperforms all baselines in terms of F1 score on our test set. We also compared our method with the baseline VAE which uses a question reconstructor  to adjust the task-specific model  via maximizing a variational lower bound of log P ( q | d ) as the regularization term L vae ( ,  ) . To pre-train the task-specific model for this method, we simply obtained the best task-specific model trained with HardEM-thres. VAE optimizes the task-specific model on L vae ( ,  ) with reinforcement learning where P  ( q | d, z ) is used as learning signals for the task-specific model. Despite our efforts to stabilize training, the F1 score still dropped to 36.28 after optimizing the overall objective L ( ,  ) for 1,000 steps. By contrast, our method does not use P  ( q | d, z ) to compute learning signals for the task-specific model but rather uses it to select solutions to train the task-specific model, which makes a better use of the question reconstructor. Text2SQL is a popular semantic parsing task. Given a question q and a table header d = [ h 1 , ..., h L ] where h l is a multi-token column, a parser is required to parse q into a SQL query z and return the execution results. Under the weakly supervised setting, only the final answer is provided while the SQL query is not. Following Min et al. (2019), Z is approximated as a set of non-nested SQL queries with no more than three conditions: Z = { z = ( z sel , z agg , { z condk } 3 k =1 ) | f ( z ) = a, z sel  { h 1 , ..., h L } , z condk  { none }  C, z agg  { none, sum, mean, max, min, count }} Brady throw? needs counting, but the related mentions are not known. (Chen et al., 2020) partly solved this problem by adding model-predicted solutions (with correct answer) into the initial solution sets as learning proceeds. In this paper, we kept the initial solution sets unchanged during training, so that different QA tasks share the same experimental setting. where z agg is an aggregating operator and z sel is the operated column (a span of d ). C = { ( h, o, v ) } is the set of all possible conditions, where h is a column, o  { = , <, > } , and v is a question span. We experimented on WikiSQL (Zhong et al., 2017) under the weakly supervised setting 2 . We chose SQLova (Hwang et al., 2019) as the task-specific model which is a competitive text2SQL parser on WikiSQL. Hyperparameters were kept the same as in (Hwang et al., 2019). We used the solution sets provided by Min et al. (2019). Results : All models in Table 5 do not apply execution-guided decoding during inference. Our method achieves new state-of-the-art results under the weakly supervised setting. Though without supervision of ground-truth solutions, our execution accuracy (i.e., accuracy of execution results) on the test set is close to that of the fully supervised SQLova. Notably, GRAPPA focused on representation learning and used a stronger task-specific model while we focus on the learning method and outperform GRAPPA with a weaker model. 0 20 40 60 80 100 62 65 68 71 74 77 [0,3) [3,5) [5,7) [7,9) [9,+) % o f D a t a F 1 S c o re |Z| % of Data HardEM-thres Ours 0 25 50 75 100 65 68 71 74 77 [0,3) [3,5) [5,7) [7,9) [9,+) % o f D a t a F 1 S c o re |Z| % of Data HardEM-thres Ours Figure 4: Performance on test examples with different size of Z on DROP. Fig 4 shows the performance on test data with different size of solution set 3 . Our method consistently outperforms HardEM-thres and by a large margin when test examples have a large solution set. The more complex a question is, the larger the set of possible solutions tends to be, the more likely a model will suffer from the spurious solution problem. We therefore investigated whether our learning method can deal with extremely noisy solution sets. Specifically, we extracted a hard train set from the original train set of WikiSQL. The hard train set consists of 10K training data with the largest Z . The average size of Z on the hard train set is 1,554.6, much larger than that of the original train set (315.4). We then compared models trained on the original train set and the hard train set using different learning methods. 2 WikiSQL has annotated ground-truth SQL queries. We only used them for evaluation but not for training. 3 In this experiment, | Z | is only seen as a property of an example. Evaluated solutions are predicted by the task-specific model but not from Z . As shown in Fig 5, models trained with our method consistently outperform baselines in terms of logical form accuracy (i.e., accuracy of predicted solutions) and execution accuracy. When using the hard train set, the logical form accuracy of models trained with HardEM or HardEM-thres drop to below 14%. Compared with HardEM, HardEM-thres is better when trained on the original train set but is worse when trained on the hard train set. These indicate that model confidence can be unreliable and thus insufficient to filter out spurious solutions. By contrast, our method explicitly exploits the semantic correlations between a question and a solution, thus much more resistant to spurious solutions. Training Epochs 2 4 6 8 10 BART base w/ HardEM 65.1 60.8 59.7 58.6 61.0 SQLova w/ HardEM 61.3 62.2 61.8 61.8 61.7 SQLova w/ Ours 79.7 82.8 79.8 81.2 87.4 Table 6: Accuracy on the SQL selection task. As we used BART base as the question resconstruc-tor, we investigated how our question reconstructor contributes to performance improvements. We first investigated whether BART base itself is less affected by the spurious solution problem than the task-specific models. Specifically, we viewed text2SQL as a sequence generation task and fine-tuned a BART base on the hard train set of WikiSQL with HardEM. The input of BART shares the same format as that of SQLova, which is the concatenation of a question and a table header. The output of BART is a SQL query. Without constraints on decoding, BART might not produce valid SQL queries. We therefore evaluated models on a SQL selection task instead: for each question in the development set of WikiSQL, a model picks out the correct SQL from at most 10 candidates by selecting the one with the highest prediction probability. As shown in Table 6, when trained with HardEM, both BART base parser and SQLova perform similarly, and underperform our method by a large margin. This indicates that using BART base as a task-specific model can not avoid the spurious solution problem. It is our mutual information maximization objective that makes a difference. We further investigated the effect of the choice of question reconstructor. We compared BART base with two alternatives: (1) T-scratch: a three-layer Transformer (Vaswani et al., 2017) without pretraining and (2) T-DAE: a three-layer Transformer pre-trained as a denoising auto-encoder of questions on the train set; the text infilling pre-training task for BART was used. As shown in Table 7, our method with either of the three question reconstruc-tors outperforms or is at least competitive with baselines, which verifies the effectiveness of our mutual information maximization objective. What's more, using T-DAE is competitive with BART base , indicating that our training objective is compatible with other choices of question reconstructor besides BART, and that using a denoising auto-encoder to initialize the question reconstructor may be benefi-cial to exploit the semantic correlations between a question and its solution. As solutions with correct answer can be spurious, we further analyzed the quality of predicted solutions. We randomly sampled 50 test examples from DROP for which our method produced the correct answer, and found that our method also produced the correct solution for 92% of them. To investigate the effect of different learning methods on models' ability to produce correct solutions, we manually analyzed another 50 test samples for which HardEM, HardEM-thres, and our method produced the correct answer with different solutions. The percentage of samples for which our method produced the correct solution is 58%, much higher than that of HardEM (10%) and HardEM-thres (30%). For experimental details, please refer to the appendix. Fig 6 compares NeRd predictions on four types of questions from DROP when using different learning methods. An observation is that NeRd using our method shows more comprehensive understanding of questions, e.g., in the Arithmetic case, NeRd using our method is aware of the two key elements in the question including the year when missionaries arrived in Ayutthaya and the year when the Seminary of Saint Joseph was built, while NeRd using HardEM-thres misses the first element. What's more, NeRd using our method is more precise in locating relevant information, e.g., in the first Sorting case, NeRd with our method locates the second appearance of 2 whose contextual semantics matches the question, while NeRd using HardEM-thres locates the first appearance of 2 which is irrelevant. These two observations can be attributed to our mutual information maximization objective which biases a task-specific model towards those solutions that align well with the questions. However, we also observed that when there are multiple mentions of relevant information of the same type, NeRd trained with HardEM-thres or our method has difficulty in recalling them all, e.g., in the second Sorting case, the correct solution should locate all four mentions of Sebastian Janikowski's field goals while NeRd using either method locates only two of them. We conjecture that this is because the solution sets provided by Chen et al. (2020) are noisy. For example, all precomputed solutions of sorting type for numeric answers involve up to two numbers from reference information, which makes it hard for a model to learn to sort more than two numbers. exploit the semantic correlations between a question and its solution via mutual information maximization. During training, we pair a task-specific model with a question reconstructor which guides the task-specific model to predict solutions that are consistent with the questions. Experiments on four QA datasets demonstrate the effectiveness of our learning method. As shown by automatic and manual analyses, models trained with our method are more resistant to spurious solutions during training, and are more precise in locating information that is relevant to the questions during inference, leading to higher accuracy of both answers and solutions. This work was partly supported by the NSFC projects (Key project with No. 61936010 and regular project with No. 61876096). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2019GQG1 and 2020GQG0005.
abstain abstain abstain abstain objective result abstain abstain abstain abstain other abstain abstain abstain objective objective abstain method method abstain abstain objective result other other other other other other other other other other other other other other other objective other other abstain abstain abstain method abstain method method abstain method method abstain method abstain method abstain abstain other method abstain other other abstain abstain abstain abstain method method method abstain abstain method method abstain method abstain method abstain abstain method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result other abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method objective result other other