Neural discrete reasoning (NDR) has shown remarkable progress in combining deep models with discrete reasoning. However, we find that existing NDR solution suffers from large performance drop on hypothetical questions, e.g.,  what the annualized rate of return would be if the revenue in 2020 was doubled . The key to hypothetical question answering (HQA) is counterfactual thinking, which is a natural ability of human reasoning but difficult for deep models. In this work, we devise a Learning to Imagine (L2I) module, which can be seamlessly incorporated into NDR models to perform the imagination of unseen counterfactual. In particular, we formulate counterfactual thinking into two steps: 1) identifying the fact to intervene, and 2) deriving the counterfactual from the fact and assumption, which are designed as neural networks. Based on TAT-QA, we construct a very challenging HQA dataset with 8,283 hypothetical questions. We apply the proposed L2I to TAGOP, the state-of-the-art solution on TAT-QA, validating the rationality and effectiveness of our approach. Neural discrete reasoning (Dua et al., 2019) is an emerging technique for machine reading comprehension (Rajpurkar et al., 2016) which aims at answering numerical questions from textual (Dua et al., 2019) or hybrid (Zhu et al., 2021) context 1 . NDR combines deep neural network with discrete and symbolic reasoning ( e.g., addition, sorting, or counting) (Dua et al., 2019) and enables the comprehension of complex contexts and compositional questions, which is critical for many practical applications such as automatic diagnosis (Wei et al., 2018) and robo-advisor (Fisch et al., 2019). Existing state-of-the-art NDR models implement the nuCorresponding author. 1 where hybrid includes textual and tabular data in this work merical reasoning process as neural network modules (Ran et al., 2019; Herzig et al., 2020; Zhu et al., 2021), e.g., a graph neural network for sorting (Ran et al., 2019; Chen et al., 2020a). In this work, we extend NDR to hypothetical question answering (HQA), where the question consists of an assumption beyond the context (Fig-ure 1). The ability of HQA will undoubtedly enhance the practical use of NDR due to the universality of hypothetical questions. However, current NDR models face severe generalization failure on hypothetical questions. An empirical evidence on such vulnerability is that the state-of-the-art model (Zhu et al., 2021) encounters a sharp performance drop (F1 score drops from 68.6% to 3.8%) on the TAT-QA dataset when changing the questions to be hypothetical by adding a related assumption (see details in Section 2, Table 3). We postulate that the failure is due to unable of imagining the counterfactual context according to the assumption (Figure 1). To pursue such reasoning ability, we resort to the concept of counterfactual thinking (Pearl, 2019) from the theory of causality, which is the ability to imagine and reason over unseen cases based on the seen facts and counterfactual assumptions. In this light, we consider modeling counterfactual thinking as neural network modules that can be seamlessly incorporated into existing NDR models. One straightforward solution is to model counterfactual thinking as a generation procedure with the fact and assumption as inputs by using a generation model such as GPT (Brown et al., 2020). However, such uncontrollable model (Zou et al., 2021) can hardly generate high-quality context for two reasons: 1) the context is more complex than plain text, which can include a table (Figure 1); and 2) NDR requires a precise context with the correct numbers (Figure 1, $132,935 for the finished goods in 2019 ). Therefore, we resort to an alternative approach: constructing the counterfactual 57 What was the change in finished goods from 2018 to 2019? by intervening on the factual context. As shown in Figure 1, the assumption changes one entry in the table, e.g., $133,682 to $132,935 . This is coherent with the causal inference theory (Pearl, 2009) where the target variable is intervened according to the hypothetical condition to infer a counterfactual. We propose Learning to Imagine, where the counterfactual thinking is implemented with two intervening steps: 1) identifying the facts to intervene, and 2) deriving the result of intervention. To pursue accurate context, we derive the intervention with a set of discrete operators such as SWAP and ADD for imagination. To evaluate the counterfactual thinking ability, we recruit volunteers with domain expertise to construct an HQA dataset based on TAT-QA (Zhu et al., 2021) by posting an assumption for each original question, named TAT-HQA. We apply L2I to TAGOP (Zhu et al., 2021), and obtain a promising solution for HQA. In summary, the main contributions are as follows:  We highlight the importance of counterfactual thinking in NDR and formulate counterfactual thinking as an intervening procedure to achieve precise imagination. We devise the L2I module, which is designed as neural network operations and can be seamlessly incorporated into the NDR model for answering hypothetical questions. We construct a challenging HQA dataset and conduct extensive experiments on the dataset, where the performance validates the rationality and effectiveness of the proposed L2I. In the general setting of machine reading comprehension, the task is to answer a question according to the facts in a given context. Formally, it is to learn a function y = f ( q , c ) , where y , q , and c are the word list representing the answer, the question, and the context 2 respectively. This work studies a new and more challenging task that focuses on hypothetical question. As shown in Figure 1, a hypothetical question includes an assumption, e.g.,  if the amount in 2019 was $132,935 thousand instead . The target of HQA is to learn y = f ( q , c , a ) where a denotes the assumption. The existence of an assumption calls for the imagination of a counterfactual context before inferring the answer, pushing the NDR model to grasp both semantic understanding and counterfactual thinking. To facilitate the evaluation of HQA and diagnose counterfactual thinking, we construct an HQA dataset based on TAT-QA (Zhu et al., 2021), which is a QA dataset with a mix of tabular and textual context extracted from financial reports. Inspired by previous work on constructing counterfactual samples (Kaushik et al., 2019), we recruit college students with finance-related majors to imagine an intervention based on the factual question and context from TAT-QA which involves numerical thinking, e.g., a change of number. Then they phrase the intervention into an assumption, forming a  what if  type of question, and calculate the answer (see an example in Figure 1). To ensure the diversity of the phrasing, annotators are free to generate various phrasing of the assumption, and there is no restriction on the position of the assumption. Usually, the assumption appears either before of after the factual question. Each hypothetical question is related to one factual question from TAT-QA, but each factual question in TAT-QA is not guaranteed to have one hypothetical question. We follow the quality control approaches of annotator training and two-round validation in TAT-QA to guarantee the quality of the hypothetical questions. Following TAT-QA, the hypothetical questions are also labeled with four answer types: arithmetic , span , count , and multi-span , three types of answer sources: table, text and table-text, and a derivation on how the answer is derived from the context. In total, we obtain 8,283 hypothetical questions, naming it as TAT-HQA. The statistics of TAT-HQA are shown in Table 1. We follow the split of training, testing and validation set of TAT-QA as shown in Table 2. We conduct a pilot study on the generalization ability of existing NDR models on hypothetical questions. In particular, we evaluate TAGOP (Zhu et al., 2021), which is the state-of-the-art model on TAT-QA (see detailed settings in Section 4.1) by training on TAT-QA and testing on TAT-HQA. In Table 3, the huge performance drop shows that even the state-of-the-art NDR model lacks counterfactual thinking ability. We aim to empower NDR models with counterfactual thinking ability. Firstly, we decide to choose the approach of explicitly modeling discrete operations, since existing NDR solutions have demonstrated its superiority (Dua et al., 2019; Ran et al., 2019; Herzig et al., 2020; Zhu et al., 2021). We devise a Learning to Imagine module to model counterfactual thinking (Section 3.1), and then incorporate the L2I module (Section 3.2) into existing NRD methods, followed by a discussion about potential extensions (Section 3.3). Functionally speaking, the L2I module aims to construct a counterfactual context based on the factual context and the assumption. We formulate it as: c (cid:48) = g ( c , a ) , where the counterfactual context c (cid:48) is the status of the context c after the assumption a is executed. Resorting to the language of causality, it can be expressed as the do -operation that intervenes a variable to execute the assumption and the action to derive the outcome of the intervention 3 (Pearl, 2009). The key to achieving counterfactual thinking in NDR lies in: 1) parsing the assumption to identify the target fact to intervene; and 2) deriving the assumed value to construct the counterfactual context. Taking the hypothetical question in Figure 1 as an example, an ideal L2I should recognize the target variable ( finished goods in 2019 ), identify the corresponding fact ( $133,682 ), and replace the fact with the assumed value ( $132,935 ). Two-step Formulation. To this end, we propose a two-step formulation of counterfactual thinking for HQA to perform the identification and derivation. Formally, Step 1: i = r ( c , a , q ) (1) Step 2: c (cid:48) i = d ( c i , c , a ) , c (cid:48) j = (cid:40) c (cid:48) j , j = i, c j , otherwise . Step 1: Identifying the target fact. r (  ) denotes the tagging function which scans the factual context c to recognize the fact related to the assumption a and the question q . i is the word position of the identified fact c i . Step 2: Deriving intervention result. d (  ) denotes the deriving function that parses the assumption a to infer the discrete operation and the premise to derive the assumed value c (cid:48) i . As to the assumption in Figure 1, the derivation requires a SWAP operation and a premise $132,935 . This step then calls for an editing operation to construct the counterfactual context c (cid:48) . Module Design. Based on the two-step formulation, we then design the L2I module as neural network operations. We have two considerations for the module design: 1) the module should recognize the semantic connection between the assumption and the context, and 2) the module should uniformly support various discrete operations to 3 Note that we adopt the do -expression (Pearl, 2009) of counterfactual. enable accurate derivation. To this end, we devise four key building blocks for the L2I module:  Encoder . It projects the raw content into latent representation. Inspired by the recent research on NDR, we employ a pre-trained language model (PLM), i.e., RoBERTa (Liu et al., 2019), as the encoder to learn an overall representation of the context, question, and assumption; H = PLM ([ CLS , c , SEP , { q , a } , SEP ]) (2) where L and M are the length of the tokenized inputs. CLS and SEP denote the beginning and the separation token of the input. { q , a } represents that the relevant position of a to p can vary. We do not assume q to always precede a due to the various location of a in the annotation. Matching block . It distills the semantic connection between the factual question, the factual context and the hypothetical assumption (Figure 1,  amount in 2019  and  $132,935 ). After applying the token-level self-attention of PLM, we aim to further distill the sequence-level semantic connection between the factual part (the question and the context) and the hypothetical part (the assumption). We obtain the factual and assumption representations by masking H according to the position of the question, the context and the assumption, which splits H into 2 nonoverlapping parts. Inspired by the success of cross-attention (Kim et al., 2018) in associating different sources, e.g., image-image (Hou et al., 2019) and image-text (Lu et al., 2019), we adopt cross-attention between the factual representation and the assumption representation, followed by self-attention respectively. Formally, the calculation of the k -th layer is, H f = mask ( H , pos ( { c, q } )) H a = mask ( H , pos ( a ))  H kf = MHA (cid:16) H k  1 f , H k  1 a , H k  1 a (cid:17)  H ka = MHA (cid:16) H k  1 a , H k  1 f , H k  1 f (cid:17) H kf = MHA (cid:16)  H kf ,  H kf ,  H kf (cid:17) H ka = MHA (cid:16)  H ka ,  H ka ,  H ka (cid:17) where MHA (  ) denotes the multi-head attention (Vaswani et al., 2017) with a triple of query, key, and value as the input. The residual connection and batch normalization are applied as the default choice. mask (  ) denotes the masking operation, and pos ( x ) is a binary vector with the same length of H denoting the positions of x in the input of PLM. Tagging head . It models the identification of target fact as a token-wise tagging. Formally, t i = (cid:40) 1 ,  ( j ) , argmax ( p j ) = 1  h Kj (cid:55) c i , 0 , otherwise . p j = softmax ( MLP (cid:16) h Kj (cid:17) ) (3) where t i is a binary tag for the fact c i . c i will be a target as at least one of its tokens is tagged. We use h Kj (cid:55) c i to represent the mapping between token and fact, which is true if token j belongs to fact c i . For each token, we employ a 2-way classifier MLP (cid:16) h Kj (cid:17) to predict its probability of being tagged as p j where argmax ( p j ) = 1 means positive (see Appendix A for more de-tails). Deriving head . It derives the intervention result for the target fact. To calculate the intervention result, we select a set of commonly used discrete operators such as SWAP , ADD , and MINUS ( cf. Appendix B). Then, we model the derivation as making a choice across the operators and tagging the premise for executing the operator. In particular, we adopt a tagging head to identify the premise and a multi-way classifier for choosing operators, which is formulated as: o = softmax ( MLP ( h CLS ) ). o  RO is a distribution over the operators where O denotes the number of operators. h CLS corresponds to the CLS token in H . Most recent NDR models (Ran et al., 2019; Andor et al., 2019; Chen et al., 2020a; Herzig et al., 2020; Zhu et al., 2021) consist of two main modules: 1) a PLM to encode the context and the question into latent representations, and 2) a reasoning module that chooses the discrete operator and identifies the operands according to the latent representations. As shown in Figure 2, we can seamlessly incorporate the proposed L2I into such NDR model as an intermediate module, which performs imagination before discrete reasoning. In particular, we simply let the reasoning module conduct operand look-up within the counterfactual context constructed by L2I. Besides, we let L2I reuse the PLM in the NDR model to reduce the model complexity and training time. Model training. Existing NDR methods typically follow the supervised learning paradigm to optimize the model parameters (Dua et al., 2019). Suppose we have a set of labeled questions D = { <  y , ( q , c , a ) > } , the training objective can be abstracted as min  (cid:80) DQA ( y , f ( q , c , a )) where  denotes model parameters. Note that QA (  ) measures the discrepancy between the ground-truth and the predicted answers which can have different formats. For instance, it can be a combination of the cross-entropy (CE) loss over the operand look-up and the CE loss over the choice of discrete operation (Herzig et al., 2020; Yin et al., 2020; Zhu et al., 2021). When applying L2I to an existing NDR method, we keep its question-answering objective unchanged. To optimize the L2I module, we incorporate supervision on the classifiers in the tagging head and deriving head. Formally, min  (cid:88) D (cid:16) QA (cid:0)  y , f ( q , c , a ) (cid:1) + 1 L (cid:88) j<L CE (cid:0)  p j , MLP ( h Kj ) (cid:1) + CE (cid:0)  o , MLP ( h CLS ) (cid:1)(cid:17) , (4) where  p j  { 0 , 1 } denotes the label of the target fact (token j in context) or the premise (token j in assumption); and  o  RO is the label of the deriving operator (see Appendix C for the details of label construction). Readers might have raised the following two concerns for L2I: 1) the operators defined are limited, and 2) the operators are tailored to one step of derivation on one target fact. Actually, it is a common approach for current state-of-the-art NDR models to apply a set of defined operators (Ran et al., 2019; Chen et al., 2020a; Zhu et al., 2021). For the first concern, by doing more fine-grained classification on the numerical reasoning process in the dataset, we can derive new operators and simply plug them into L2I. Note that the annotation of numerical intervention of TAT-HQA does not follow the defined operators in Appendix C, but the operators are summarized from the data. Our defined operators can cover over 90% of the training data. For the second concern, we discuss two potential solutions by our L2I framework, and we leave the implementation as future work. Multi-fact intervention. The assumption a can include intervening multiple facts, e.g.,  if the Finished goods in 2018 and 2019 were both doubled . Apparently, if the target facts are independent, we can easily handle such an assumption by executing L2I in multiple iterations. In other cases, L2I needs to recognize the relationship among the target facts. If such relationship is available, L2I should be able to handle such cases as the corresponding multivariable operator is added to the deriving head. Multi-iteration derivation. In causal inference, a rigorous derivation of an intervention considers the successors of the target variable, e.g., finished goods in 2019 affects total inventories in 2019 . Currently, we omit the following iterations in Step 2 of L2I ( cf. Eq 1). This is because not all successors are necessary for answering the question. For instance, answering the question in Figure 1 does not require the post-intervention value of total inventories in 2019 . In conventional causal inference, such successors will also be omitted according to the local surgery principle (Pearl, 2009). Moreover, we believe that the following iterations can be achieved by the current L2I module in an iterative manner. Assume that NDR model equipped with L2I can answer the hypothetical questions requiring one-iteration derivation ( i.e., c i  c (cid:48) i ). We can thus derive the value of successors ( e.g., c (cid:48) i  c (cid:48) j ) by forming a simple hypothetical question:  What c j would be if c i is c (cid:48) i ?  and answering it with the NDR model. We conduct experiments on TAT-HQA dataset to answer the following questions: RQ1: How does L2I perform on HQA? RQ2: What factors influ-61 Table 4: Performance of compared methods on the TAT-HQA dataset. ence the effectiveness of L2I? Following Dua et al. (2019) and Zhu et al. (2021), we evaluate the performance with two commonly used metrics: Exact Match (EM) and numerically-focused F 1 score, where higher value (in [0, 100]) means better performance. We tune the hyper-parameters on the validation set, and report the average test performance of five different runs. Compared methods. To validate the effectiveness of our proposed L2I module, we apply it to TAGOP, obtaining an NDR model for HQA, named TAGOP-L2I. In addition to the vanilla TAGOP, we compare our method against representative methods of traditional QA, numerical QA, tabular QA, and hybrid QA. Besides, we want to select baselines that are effective for learning counterfactual samples. The baselines are: BERT-RC (Devlin et al., 2019), a traditional QA method that selects answer spans from the context. NumNet+ V2 (Ran et al., 2019), a numerical QA method with numerically-aware graph neural network. TAPAS-WTQ (Herzig et al., 2020), a tabular QA method that focuses on parsing and understanding tables, pre-trained over tables collected from Wikipedia before training on TAT-HQA. HyBrider (Chen et al., 2020c), a hybrid QA method that considers the connection between the table and text. TAGOP , a hybrid QA method that performs discrete reasoning over both the tabular and textual contexts. It is the state-of-the-art method on TAT-QA dataset. TAGOP-CLO , incorporating the Contrastive Learning Objective (CLO) into the training objective of TAGOP, which is shown to be effective in learning the relationship between factual and counterfactual samples (Liang et al., 2020). Parameter settings. We implement TAGOP-L2I based on TAGOP 4 . We set the number of cross-attention layers to 3, and fine tune from TAGOP trained on TAT-QA with a learning rate of 5e-5, batch size of 32, and gradient accumulation step of 4. All compared methods are initialized with 4 https://github.com/NExTplusplus/ TAT-QA . the model trained on TAT-QA and then fine-tuned on TAT-HQA. For TAGOP-CLO, we conduct max pooling for H and adopt cosine similarity as the distance metric. We select the corresponding factual question as the positive sample and a randomly selected factual question as the negative sample. The weight for the contrastive loss is 0.1. Overall performance. Table 4 shows the performance of the compared methods on the TAT-HQA dataset. We can observe that: 1) TAGOP-L2I achieves the best performance among all the compared methods. In particular, it outperforms the best baselines by 19.8% and 19.7% on EM and F 1 , respectively. Such significant performance gain validates the effectiveness of the L2I module and reveal the rationality of modeling counterfactual thinking as a neural network module. 2) TAGOP-CLO outperforms TAGOP by 10.5% and 10.4% on EM and F 1 . The only difference between these two methods is that TAGOP-CLO incorporates an extra CLO. The improvement indicates that learning the relationship between the factual and counterfactual samples with CLO provides some clue for counterfactual imagination, yet it is still worse than directly learning to imagine with neural network modules. 3) As to the remaining methods, their performance has a clear gap between TAGOP, which is consistent with the result on the TAT-QA dataset (Zhu et al., 2021). This is because both datasets have textual and tabular texts, where the ability of TAGOP to perform discrete reasoning across hybrid contexts brings significant advantages. 4) The performance achieved is still low w.r.t. the two metrics ( e.g., 54.4  100), showing a large space for future exploration on the challenging TAT-HQA dataset. Detailed performance. To further investigate the effectiveness of the proposed L2I module, we perform a detailed comparison between TAGOP-L2I and TAGOP w.r.t. the discrete operation required in answering the question or counterfactual thinking. We group the questions according to 1) the answer type and 2) the operator to derive the intervention. Table 5 shows the group-wise 62 Table 5: Detailed performance of TAGOP-L2I and TAGOP w.r.t. answer type and deriving operator type. performance. As to answer type (the left half), we have the following observations: 1) TAGOP-L2I outperforms TAGOP on all groups, showing the superior ability of learning to imagine to all types of questions. 2) Particularly, on the arithmetic group, which is also the largest group ( cf. Table 1), TAGOP-L2I largely outperforms TAGOP. For this group, the key difference between TAGOP-L2I and TAGOP is whether the derivation of intervention and calculation of the answer are achieved by separate modules. The superior performance of TAGOP-L2I validates the rationality of modeling counterfactual thinking as a separate module. It should be noted that the separation also facilitates the generalization to new operations since the modules can be separately updated. 3) The performance of TAGOP on arithmetic has a large gap with other types, showing that arithmetic questions are more difficult to conduct imagination and reasoning even though arithmetic makes up the majority of TAT-HQA data. As to TAGOP-L2I, the gap between arithmetic question and other types of question largely reduces, validating the effectiveness of learning intervention with discrete operators and neural network modules. As to operator types (the right half), we observe that: 1) TAGOP-L2I achieves imagination on the majority of operator types with better performance than TAGOP, yet TAGOP can only achieve imagination on a few operator types. The better performance of TAGOP-L2I is attributed to modeling the deriving operations as specific operators. We thus believe that TAGOP-L2I can generalize well to more deriving operations by simply incorporating the operators, as long as the corresponding training questions are not rare. This result thus reflects the advantage of the unified operator framework adopted by the L2I module, which is consistent with previous work (Andor et al., 2019). 2) Across the groups, TAGOP achieves relatively good performance on the SWAP group, which replaces the target fact with a number in the assumption. It corresponds to the simplest imagination since the assumed value ( i.e., c (cid:48) i ) is explicitly mentioned in the assumption. Therefore, the result shows that the NDR model can achieve simple counterfactual thinking by learning to answer hypothetical questions. However, such indirect guidance on imagination fails on the groups requiring more complex imagination, e.g., requiring add or minus. 3) TAGOP-L2I achieves the worst performance on SWAP MIN NUM , which is merely comparable to TAGOP. We suspect the reason is that the operation of SWAP MIN NUM is very close to SWAP , which may confuse the deriving head when making classification over the operators. To address this issue, it is worth considering the operator relation in the deriving head in the future. Study on L2I module design. We then explore the influence of network architecture on the effectiveness of the L2I module from three perspectives: 1) module depth; 2) configuration of the matching block; and 3) the setting of PLM. Figure 3(a) shows the validation result of TAGOP-L2I as increasing the matching block from 1 to 4 layers. We can observe that: 1) Stacking more layers does not always bring performance gain. 2) In particular, three layers of matching block achieve the best performance on TAGOP-L2I. The result indicates that three layers should be sufficient to capture the semantic connection across the context, question and assumption. This is reasonable since the average length of both assumption and question are only around 10 words ( cf. Table 2). As to the architecture of the matching block, we evaluate three variants from the default choice ps, self-a which enables parameter sharing across layers ( i.e., ps ) and applies both cross-MHA on the factual and assumption representations and self-MHA for each of them ( i.e., self-a ). The three variants are: 1) p-s, w/o self-a , which removes self-MHA; 2) w/o p-s, self-a , which disables parameter sharing; and 3) w/o p-s, w/o self-a , which adopts both changes. Figure 3(b) shows the performance of the four versions of TAGOP-L2I with 63 Figure 3: Performance of TAGOP-L2I under difference module configurations. three layers of the matching block. From the figure, we can observe that: 1) The default choice largely outperforms the variants, validating the rationality of our module design. 2) Disabling parameter sharing hinders the counterfactual thinking, which indicates that keeping the same parameters through the process of matching factual and assumption representations is beneficial for extracting the semantic correlation. 3) Removing self-MHA also leads to sharp performance drop, which justifies the contribution of self-MHA in the L2I module. It is thus essential to also separately process the semantic information of the factual and the assumption representations in the matching block. We also conduct experiments on fixing the parameter of PLM during training on TAT-HQA as initialized by TAT-QA. The performance drops to EM 48.5 and F 1 49.0. Fixing the parameter of PLM largely impedes the performance of TAGOP-L2I on TAT-HQA, showing that encoding factual and hypothetical questions requires different mechanisms. To further investigate the difference in answering factual and hypothetical questions, we test TAGOP-L2I on TAT-QA. The result in Figure 4 shows that training on TAT-HQA causes a performance drop in counting , span and multi-span groups of TAT-QA, and performs similar on the in arithmetic group. We conjecture the performance drop in the first three groups is because the question-answering label in TAT-HQA under the same c and q is different from TAT-QA. However, for arithmetic questions, the question-answering label for one pair of c and q remains the same between TAT-HQA and TAT-QA, and the intervention is achieved explicitly by Figure 5: Group-wise performance of TAGOP-L2I and TAGOP-L2I-T w.r.t. operator type. deriving operators and tagging head. Study on L2I training objective. We then investigate the influence of imagination-oriented training objectives on the effectiveness of L2I. In particular, we evaluate a variant TAGOP-L2I-T trained only with the question-answering objective ( i.e., QA (  ) ). That is, TAGOP-L2I-T learns to implicitly imagine the final answer. Figure 5 shows the group-wise performance of TAGOP-L2I and TAGOP-L2I-T w.r.t. the type of operator for deriving the intervention. We can observe the followings. 1) On most groups, TAGOP-L2I largely outperforms TAGOP-L2I-T, demonstrating the rationality of learning to imagine explicitly. 2) On SWAP group TAGOP-L2I-T achieves comparable result to TAGOP-L2I. As SWAP is the simplest deriving operator, the result shows that the implicit guidance can achieve simple imagination, yet is still less effective than the explicit manner. 3) TAGOP-L2I-T achieves better performance on SWAP MIN NUM group. As SWAP MIN NUM is a rare operator ( cf. Table 6) and involves the most complex imagination process ( cf. Appendix B), we conjecture that learning complex operators is more difficult than implicitly learning. This may shed light on the rules of deriving new operators that simple operators with ample training data is preferred over complex operators with less training data. Counterfactual thinking. Existing research incorporates counterfactual thinking into deep models from two main perspectives: counterfactual training and counterfactual inference . Counterfactual sample has become an emerging data augmentation technique in computer vision (Chen et al., 2020b) and natural language processing (Kaushik et al., 2019) to enhance model robustness. For instance, the technique is applied in visual QA (Chen et al., 2020b; Agrawal et al., 2018; Agarwal et al., 2020; Gokhale et al., 2020), vision-language navigation (Fu et al., 2020; Par-vaneh et al., 2020), table entailment (Eisenschlos et al., 2020), sentiment analysis (Kaushik et al., 2019; Yang et al., 2020), natural language inference (Kaushik et al., 2019), named entity recognition (Zeng et al., 2020), and dialogue system (Zhu et al., 2020). Along this line, a series of studies explore how to maximize the effect of counterfactual samples by combining with different learning paradigms, such as adversarial training (Zhu et al., 2020; Fu et al., 2020; Teney et al., 2020), contrastive learning (Liang et al., 2020), causal graph (Gokhale et al., 2020), posterior regularization (Ramakrishnan et al., 2018), and designing new learning paradigms (Gokhale et al., 2020). A few studies along this line also generate counterfactual samples with neural networks (Sauer and Geiger, 2021; Yue et al., 2021). They are inherently different from our work due to their reliance on causal graph and the causal expression of the hypothetical condition for improving robustness. Moreover, they supervise the generation with other related tasks such as image classification. In contrast, we formulate imagination as an explicit learning objective, i.e., learning to imagine. Additionally, in commonsense reasoning, counterfactual samples are also utilized through hyperbole generation (Tian et al., 2021), story generation (Qin et al., 2019) and commonsense QA(Huang et al., 2019), which is also a related yet different strand of research. Another line of research performs counterfactual inference over the predictions of deep model to incorporate counterfactual thinking (Yue et al., 2021; Wang et al., 2021; Niu et al., 2021; Tang et al., 2020). However, they perform counterfactual inference according to causal graph which is not available in NDR tasks. Neural discrete reasoning. Recent research on NDR focuses on enhancing the discrete reasoning ability of deep models in two main directions: reasoning with more discrete operations (Dua et al., 2019; Ran et al., 2019; Chen et al., 2020a) and reasoning over more complex context . For instance, NumNet (Ran et al., 2019) and QDGAT (Chen et al., 2020a) leverage graph neural network to enhance comparison oriented operations. GenBERT (Geva et al., 2020) uses pre-trained language models to generate the numerical answer, which breaks the limitation of fixed operators. NMN (Gupta et al., 2019) and FinQA (Chen et al., 2021) model the discrete reasoning process as executing programs. As to extending the context, several studies try to enable the NDR model to operate on context with semi-structured tabular data and hybrid data (Chen et al., 2020c; Herzig et al., 2020; Chen et al., 2021). Our paper studies the hybrid data, yet extends the scope of NDR to hypothetical questions. Moreover, beyond the ability of discrete operations, the main idea is to endow NDR models with the ability to think counterfactually. In this work, we pointed out a key issue of existing NDR models: lacking counterfactual thinking. We proposed an L2I module, which can imagine the counterfactual according to a textual assumption. By applying the proposed module in the NDR model, we enable the model to answer hypothetical questions. We constructed a HQA dataset and conducted extensive experiments on the dataset, which validates the effectiveness of our method. This work opens up a new research direction about modeling counterfactual thinking through neural network. In the future, we will further extend the L2I from the following perspectives: 1) handling of multiple interventions; 2) rigorous derivation of intervention with consideration of successors; 3) incorporation of the relations across the deriving operators; and 4) construction of complex operators by dynamically combining basic operators. Moreover, we will explore the translation between assumptions in natural language and causal expression to further connect the L2I framework with conventional causal theory, and facilitate automatic causal inference with neural network. This work is supported by Sea-NExT Joint Lab, Singapore MOE AcRF T2, and Natural Science Foundation of China (Grant No. U21B2026).
abstain result abstain method method method objective abstain abstain abstain abstain objective abstain abstain abstain abstain result method abstain abstain method abstain abstain abstain objective method method result objective method objective abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain other abstain method method abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain result abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain other other other other other other abstain other objective other other other other other other other other other objective other abstain objective objective method objective objective objective other