Transformers have been shown to be able to perform deductive reasoning on a logical rulebase containing rules and statements written in natural language. Recent works show that such models can also produce the reasoning steps (i.e., the proof graph ) that emulate the model's logical reasoning process. Currently, these black-box models generate both the proof graph and intermediate inferences within the same model and thus may be unfaithful. In this work, we frame the deductive logical reasoning task by defining three modular components: rule selection, fact selection, and knowledge composition. The rule and fact selection steps select the candidate rule and facts to be used and then the knowledge composition combines them to generate new inferences. This ensures model faithfulness by assured causal relation from the proof step to the inference reasoning. To test our framework, we propose FAIRR (Faithful and Robust Reasoner) where the above three components are independently modeled by transformers. We observe that FAIRR is robust to novel language perturbations, and is faster at inference than previous works on existing reasoning datasets. Additionally, in contrast to black-box generative models, the errors made by FAIRR are more interpretable due to the modular approach. 1 1 Introduction The field of AI has long pursued the goal of building systems that can automatically reason over some given explicit knowledge to generate conclusions and provide the reasoning steps involved in the process (McCarthy, 1959; Newell and Simon, 1956). Recently, Clark et al. (2020) proposed a modern version of this problem, where the formal representation of knowledge is replaced by natural language statements in English. Further, 1 The source code of FAIRR has been made available at https://github.com/INK-USC/FaiRR . they proposed a transformer-based model (Vaswani et al., 2017) RuleTaker, that can predict if a candidate statement is entailed by the natural language statements, by emulating deductive reasoning. As shown in Figure 1, in this deductive reasoning task, facts and rules from the rulebase are combined iteratively to generate intermediate inferences which eventually entails the statement. Note that the reasoning process implicitly involves two steps: determining which rules and facts to combine at each iteration, followed by using them to generate an intermediate conclusion. While RuleTaker focuses on just predicting the statement entailment, some recent works (Saha et al., 2020; Tafjord et al., 2021) have further developed systems that can also generate the reasoning steps (i.e., proof graph generation ). However, these systems do not explicitly ensure the causality from the rule/fact selection to generating the intermediate inferences. Since these systems are inherently black-box models, it is unclear if such constraints are implicitly learned by the models without being enforced externally. This, in turn, questions the faithfulness of the model's internal reasoning process (Lipton, 2018). Because the model has access 1075 to the full theory at input, it might use additional parts of the theory, than just the predicted proof, to generate the inference. In this paper, we address these shortcomings by developing a modularized framework to solve the deductive reasoning task. While existing methods generate both proofs and conclusions in a single step, in our framework we break this process into three steps: rule selection, fact selection, and knowledge composition. The rule selection step decides the relevant rule to use for an iterative inference step and fact selection uses this rule to select the relevant facts. Then, the knowledge composition step reasons using only the selected rule and facts to generate the next intermediate inference. In Figure 2, we show the model schematics for our system and contrast it with previous methods. Notably, we strictly restrict the information accessible at each step of our framework to make the reasoning process more faithful. For example, the fact selection step depends only on the selected rule, instead of all the rules in the rulebase. Additionally, the generated inference depends explicitly on the selected rule and facts, as opposed to all the rules and facts in prior works. This makes the proof graph a by-product of the selection steps as we don't need to generate any separate proofs. Since we constrain the inputs to each step, this also makes each subproblem easier to learn, leading to an overall more robust reasoning model. To model these three steps, we develop FAIRR, in which each component is a transformer-based model learning to perform the modular tasks. Specifically, we use RoBERTa-based models (Liu et al., 2019) for the two selection tasks and a T5-based model (Raffel et al., 2020) for the composition task. Similar to ProofWriter, we use synthetic rulebases to train FAIRR. To test the deductive reasoning capabilities in a more comprehensive way, we experiment with both existing deductive reasoning datasets and multiple newly-generated robustness dataset variants. Overall, we find that FAIRR is more robust to novel language perturbations than baselines. Additionally, our model is up to three times faster at inference due to the constrained input and outputs of different modules. Lastly, we find that the errors made by our model are more interpretable and easier to debug compared to baseline generative models. This further demonstrates the faithfulness of our modularized reasoning framework. Notations A theory T consists of a set of facts F = { f 1 ,f 2 ,...,f n } and rules R = { r 1 ,r 2 ,...,r m } expressed in natural language. An example of a theory is depicted in Figure 1. Here, the sentences in the blue and yellow boxes are facts and rules, respectively. Further, a proof graph is a directed graph connecting facts and rules that describe how a specific inference can be obtained from the theory. In Figure 1, the proof graph shows the steps involved in generating the inference  Charlie is white. . To generate the proof graph we may need to infer some intermediate conclusions c i . These inferences are considered as part of the extended facts in the theory. For example, in Fig. 1,  Charlie is kind  is an intermediate inference required to generate the correct proof graph. Deductive Reasoning The task of deductive reasoning is described as follows: given a theory T , and a statement s , predict if the theory supports the statement ( entailment prediction ) and if so, generate the proof graph that supports the statement ( proof generation ). For the example theory and statement in Figure 1, we see that the statement is indeed entailed by the theory and the valid proof graph is shown for the same. The main goal of this task is to evaluate if a model can generate valid rea-1076 soning chains in the form of proof graphs to justify its entailment prediction. Reasoning Robustness We consider an auxiliary task that evaluates the robustness of the reasoning abilities used by the model. Let P be a perturbation function that modifies a given theory T (statement s ) to a theory T  (statement s  ), such that ( T  , s  ) just has some surface changes in the natural language form but still requires the similar reasoning process as required for ( T, s ) . A function that alters the subjects in the theory to unseen subjects is an example of such perturbation function. We perturb each theory statement pair ( T, s ) to create an equivalence set defined as the set E ( T,s ) = {( T  1 , s  1 ) . . . ( T  N , s  N )} , where each ( T  k , s  k ) is derived by perturbing the original theory, and N is the total such perturbations per theory. Note that it is possible to generate different ( T  k , s  k ) pairs by controlling the stochasticity of P . The main goal of this task is to evaluate the consistency of the model's predictions with minimal variations in the input theory. Evaluation Protocol We consider three main aspects for evaluating the model performance in our study: (1) Entailment accuracy measures how accurately the model is able to predict the true statement entailment. (2) Proof accuracy measures how accurately the model can predict a valid proof for the statement. Following Saha et al. (2020); Tafjord et al. (2021), we use the strict metric for proof evaluation, i.e., for a match to count, both the predicted proof should exactly match a gold proof and the entailment should be correctly predicted. (3) Consistency measures if the models are consistent in the entailment and proof prediction for different perturbation functions. For a theory statement pair ( T, s ) and its corresponding equivalence set E ( T,s ) , consistency is defined as C = 1 N  Nk = 1 1 [ f ( T, s ) = f ( T k , s k )] , where f (  ) is the model's prediction. We compute the average consistency for both entailment and proof predictions on an equivalence set and further average across the dataset to report the consistency. As illustrated by the example in Figure 1, to reliably generate a proof graph through deductive reasoning, a model needs to generate multiple one-hop intermediate conclusions. This is the major limitation of models that use the theory to directly predict the proof (Figure 2 (a)), thus questioning the trustworthiness of the reasoning process. Next, it is also intuitive to see that in order to faithfully generate these intermediate inferences, a model should first determine the proof (i.e., know the rules and facts to use) and then use them to infer the conclusion. That is, there is a causal relation from determining the proof to then generating the conclusion. We note that ProofWriter (Iter) lacks in this aspect. As shown in Figure 2 (b), it first generates the conclusion and then the corresponding proof. Motivated by these points, we propose our causal reasoning framework which breaks the reasoning process into three desirable steps. As shown in Figure 2 (c), in our framework, first a rule r is selected using the rules and facts in the theory. Following that, some relevant facts are selected from the fact list based on the selected rule r . This step does not use the other rules R \{ r } in the theory. Finally, the selected rule and facts are jointly used to generate a new conclusion c i . In this framework, the one-step proof is explicitly determined first via the selection steps followed by the inference generation, making the proof a by-product of the whole process. In contrast, prior works learned to generate the proof along with intermediate conclusion. At a high level, FAIRR is an iterative model in which the one-hop intermediate conclusions are generated step-by-step. To model our framework described in Sec. 3.1, we have four components in FAIRR as follows. Rule Selector (RS) The rule selector is a RoBERTa-based (Liu et al., 2019) classification model that takes the concatenated statement, facts, and rules as input, and selects a rule that is used to generate an intermediate conclusion in the current iterative step. It takes the input of the form [ CLS ] s [ SEP ] F [[ SEP ] r i ] m [ SEP ] , and generates a one-hot output vector by classifying the token embedding from the [CLS] token and [SEP] tokens in front of the rules, via a linear classifier layer. Each classification is a binary classification, but overall only one of the tokens has the positive class. Here s denotes the statement, F is the facts and concatenated with any intermediate conclusions generated in a prior iteration, and { r i } denotes the i th rule in the theory that contains a 1077 total of m rules. [ ] m denotes continued concatenation. An example input and output of the rule selector is shown in Figure 3. If a [SEP] token is selected, we select the rule sentence following the corresponding [SEP] token, otherwise if the [CLS] token is selected, we decide to stop the iteration. That is, the [CLS] selection acts as a stop signal for our iterative model. We note that it is possible to have more than one likely candidate rule since there can be multiple one-hop inferences possible for a given theory. Following Tafjord et al. (2021), we randomly select one of the possible candidate rules at each iteration. Fact Selector (FS) The fact selector is RoBERTa-based (Liu et al., 2019) token classification model that takes the statement, the rule selected by the rule selector, and facts in the theory, and then predicts a set of candidate facts that can be used with the rule to generate an intermediate conclusion. It takes the input of the form [ CLS ] s [ SEP ] r [[ SEP ] f i ] n [ SEP ] , where s is the statement, r is the selected rule, and { f i } is the i th fact in the theory containing n total facts. Note that facts also include any previously generated intermediate conclusions. [ ] n denotes continued concatenation. The output is generated by classifying each [SEP] token embedding in front of a fact using a linear layer, to determine if the corresponding fact is selected or not. An example input and output for the fact selector is depicted in Figure 3. We note that it is possible to have some rules that can reason over multiple facts jointly to generate a conclusion. An example of such a rule is  rule2  in Figure 1. Hence, this component has the ability to select multiple facts. Knowledge Composer (KC) The knowledge composer is a generative text-to-text transformer T5 (Raffel et al., 2020) (T5-large) that can compose a set of facts and a rule to output a novel conclusion. The input to the model is the selected facts and rule concatenated together, and the output is the intermediate conclusion. An example input and output for knowledge composer is shown in Fig. 3. Solver The final component is the solver that operates after all iterations have finished (i.e., once the rule selector selects the [CLS] token indicating to stop the iterative inference generation pro-cess). Similar to ProofWriter, our solver currently searches for the statement in the generated intermediate inferences (string matching). If found, it [CLS] s [SEP] f1 f2 f3 [SEP] r1 [SEP] r2 [SEP] 0 1 0 Rule Selector [CLS] s [SEP] r1 [SEP] f1 [SEP] f2 [SEP] f3 [SEP] 1 1 0 Fact Selector f1 f2 r1 <eos> c1 Knowledge Composer Figure 3: Overview of components of FAIRR The rule selector and fact selectors are classification models whereas the knowledge composer is a generation model. predicts that the statement is entailed by the theory. It also search for the negation of the statement 2 , and if found, it predicts not entailed. If none of these are present, it predicts Unknown since it cannot prove or disprove the statement. The proof graph is constructed by using the one-hop proofs generated by the selected rule and facts at each step. For example, in Figure 1, the red dotted boxes (one-hop proofs) are stitched together to assemble the complete proof. For cases where the entailment is Unknown, the proof returned is None, since no proof for the statement exists in the theory. We note that our solver is not a learnable module. Each component of our model (except the solver, which is deterministic) is trained separately. We use the same dataset as ProofWriter to train these models, but process it such that each model receives only the relevant inputs according to our causal framework. More concretely, suppose for a given theory T = R + F , a possible intermediate inference is c obtained by using a rule r and a fact f . Then, a training instance of ProofWriter, which is a T5 (Raffel et al., 2020) model, uses the input { R, F } and output { c, r, f } . We process the same 2 Following ProofWriter, we perform regex to add/remove not which suffices for this dataset. instance to generate three training instances, one for each of rule selector, fact selector, and knowledge composer, respectively, as follows: RS Input = { R, F } ; RS Output = { r } , F S Input = { r, F } ; F S Output = { f } , KC Input = { r, f } ; KC Output = { c } . Our selector models have the statement s as input to the model. Also, the outputs of rule selector and fact selectors are converted to class labels instead of text since our selectors are classification models. We use cross entropy loss to train the rule selector, and binary cross entropy loss to train the fact selector. The knowledge composer is trained on language modeling loss. At inference time, the rule selector selects a rule to be used for generating one-step conclusions. Then, the fact selector selects some facts based on the selected rule, which is then collectively passed on to the knowledge composer to generate a conclusion. This three-step pipeline is run iteratively until the rule selector predicts a stop signal by selecting the [CLS] token which exits the iteration. Once the iteration finishes, the solver uses the generated intermediate inferences to decide if the statement is entailed or not, and generates a proof accordingly. Remark on Computational Complexity A practical limitation of ProofWriter is that it performs an exhaustive forward search by enumerating all possible inferences from a given theory. This leads to redundant inferences being generated for proving a particular statement. Additionally, using a text-to-text transformer model adds to the problem since it is usually quite expensive to run at inference time. In FAIRR, we alleviate this by introducing two changes. First, our causal framework allows only selected rule and facts as input to the knowledge composer, thus restricting the input length significantly. Second, augmenting the question to our selector inputs helps reduce the candidate space because these models can learn to prioritize the selection based on the relevance to both the question and the theory. This ensures that FAIRR does not perform an exhaustive forward search and prioritizes generating relevant inferences over the others. Both these changes lead to an overall improvement in inference speed. We perform more quantitative analysis on this later in Section 5.3. Datasets Following (Tafjord et al., 2021; Clark et al., 2020), we use the D* datasets for our experiments. These are a set of multiple datasets namely D0, D1, D2, D3, D0-D3, and D5. The theory in these datasets are synthetically generated with increasing reasoning depths. For example, D3 dataset contains statements that require at most 3-hop reasoning steps. The D0-D3 contains all theories in D3 plus  20% of the D0-D2 training set theories. We also use the ParaRules dataset (Clark et al., 2020) that contains around 2k theories expressed in paraphrased natural language. Additionally, we generate three datasets that evaluate the robustness of the reasoning models as follows:  Subject robustness : Here, subjects in a theory are perturbed by using some out-of-distribution proper and common names. For example, in Figure 1,  Charlie  can be replaced with  Paul  which is not used in the D* datasets. We generate five new theories corresponding to each theory of the D3 dataset, by repeatedly perturbing all the proper and common names in the theory. Attribute robustness : Here we sample out-of-distribution attributes. For example,  blue  in Figure 1 can be replaced with  soft . As above, we generate five new theories for each theory of the D3 dataset. Subject+Attribute robustness : This is a combination of subject and attribute robustness to study model performance when most of the training vocabulary is replaced by out-of-distribution words. Each theory has both novel subject and attribute. We include more details on the perturbation sets used in our experiments in Appendix B. Baselines We compare FAIRR with two variants of ProofWriter (Tafjord et al., 2021): All-at-once (PW (All)) and Iterative (PW (Iter)), wherever applicable 3 . The PW (All) model is trained to predict the entailment and generate proof graph directly from the theory and statement in a single step. The PW (Iter) generates one-step inferences and corresponding proofs iteratively, until all possible inferences are generated, and then stitches the proof 3 The code to reproduce numbers of ProofWriter is not publicly available. graph similar to our method. If not mentioned otherwise, ProofWriter uses a T5-large (Raffel et al., 2020) model. We omit comparisons with PRover since it was trained on a different dataset that adds specific constraints on the proof graph. Please refer to Appendix J for more details. We compare FAIRR with ProofWriter variants on three settings: generalization on D* datasets, robustness to perturbed theories, and efficiency in inference computation. We further conduct qualitative analysis to understand the inference errors. In this setting, we train and test both models on D0-D3 dataset. Note, D0-D3 contains statements with reasoning depths up to 3. This compares the ability of the models to generalize to seen reasoning depths at train time. The results with increasing depths of reasoning are shown in Table 1. Here, depth N/A refers to statements that cannot be proven and hence don't have an exact proof depth associated with it. We observe that overall both FAIRR and ProofWriter (Iter) performs comparably (last row with depth 'All'). Further, we find that our model's performance is lower on d = 3 , indicating that our models tend to perform weaker with increasing depths. This happens majorly because the rule selector in FAIRR tends to incorrectly select the [CLS] token to indicate a stop signal instead of generating more possible intermediate inferences. We discuss more about this in Sections 5.3 and 5.4. Please refer to Appendix C for more results on unseen reasoning depths. In this section, we test the robustness of ProofWriter (Iter) and FAIRR on different perturbed theories. Since FAIRR focuses on making deductive reasoning more robust and faithful, performance on these robustness experiments are the main results of our work. As described in Section 4, we test the robustness on three different perturbations: subject, attribute, and subject+attribute. We compare the performance of both models after training on D0-D3 dataset. The consolidated results are shown in Table 2 and depth-wise results for subject robustness are shown in Table 3. We report the entailment accuracy, proof accuracy, and consistency as defined in Section 2. Please refer to appendix D for the depth-wise breakdown of all the datasets. We observe that on subject and subject+attribute robustness, our models are consistently better than ProofWriter whereas on attribute robustness both models perform similarly. Further, we find that on average, FAIRR is both more accurate and consistent than the baseline. From this, we conclude that our model relies less on spurious correlations based on the subject while both models likely suffer from similar issues on attribute perturbations. Since ProofWriter uses the 1080 theory to generate the intermediate conclusion and proofs, it has the capacity to exploit some spurious patterns that can inflate performance. In contrast, our causal framework restricts this capacity by constraining the inputs to each component as described in Section 3.1. Hence, these robustness evaluations demonstrate one of the prime benefits of our causal and modular approach. Here we perform several analyses to evaluate the computational benefits of our method as described in Section 3.3. Inference efficiency is an important aspect of this problem for real-world scenarios where compute can be limited. Relevance of generated inferences Here, we study the relevance of the intermediate inferences generated by FAIRR and ProofWriter (Iter). Let T be the set of intermediate inferences required for generating the proof graph for the statement. Further, let G be the set of intermediate inferences actually generated by a model. Then, the precision and recall are defined as P =  T  G   G  , and R =  T  G   T  In Figure 4, we plot the precision and recall for both FAIRR and ProofWriter (Iter) with increasing reasoning depths. We find that our model has close to 1 . 0 precision at all depths, whereas ProofWriter has low precision. This demonstrates that our model is able to successfully prune the candidate inference space to generate relevant candidate inferences almost perfectly. In contrast, we see that with increasing depths, our model's recall reduces from close to 1 . 0 to  0 . 95 whereas ProofWriter has a perfect recall at all depths. While the drop is not very drastic, it indicates that our model fails to generate some essential inferences at higher depths. This is mainly because our rule selector decides to stop early and not generate further relevant inferences for some provable statements. Overall, we conclude that FAIRR always generates inferences that are relevant to solving the instance, although at higher depths it can miss some relevant conclusions. Performance under inference budget constraints We analyze the performance of FAIRR and ProofWriter under a fixed inference budget constraint by restricting the total number of conclusions that can be generated. We perform this analysis for different reasoning depths and depict the results in Figure 5. We observe that FAIRR con-Depth 0.00 0.25 0.50 0.75 1.00 1 2 3 4 5 ProofWriter (''Iter'') FaiRR sistently outperforms ProofWriter on lower budgets. This shows that FAIRR performs a prioritized generation of conclusions that are relevant to the statement, which can be useful in scenarios with limited inference budgets. See Appendix G for more comparisons. Inference runtime analysis We next compare the time taken by both the models to solve the complete D5 dev set. Although FAIRR has three separate modules that run sequentially, it is 3 . 5 times faster than ProofWriter (Iter) at inference time on average. We attribute this to the reduced inference candidate search space due to question augmentation, and smaller input size to the T5 component (refer to Section 3.3 for details). Please refer to Appendix H for more details. We further analyze the different errors made by FAIRR and ProofWriter (Iter) on 50 randomly sampled errors for each model, from the D0-D3 and the subject robustness dev splits. We manually inspect the proof inferences and compare it with the gold proof to classify the failures. The errors are broadly categorized as follows: Early stop errors: This is the most frequent error type for both models, accounting for 80% and 50% errors in FAIRR and ProofWriter, respectively. This occurs when a model incorrectly gen-1081 Input Output s 1 : If someone is blue then they are quiet. erates the stop signal and fails to generate all the required inference to prove a statement. We find that our model makes the majority of the mistakes due to early stopping. This can be possibly fixed by improving the rule selector architecture to better model the stop criteria. Wrong inference: This is the second error type, where the inferred conclusion is incorrect based on the predicted proof. This accounts for 20% and 30% errors in FAIRR and ProofWriter, respectively. We observe that our knowledge composer is makes lesser errors on average compared to the ProofWriter generative model. Other generation errors: ProofWriter makes around 20% errors where the model generated output does not make sense. For example, it can hallucinate facts that are not present in the theory. Such errors are not interpretable and questions the model's inner-working. FAIRR shows no such error, since the proofs are always interpretable in our model due to the causal framework. Overall, we find that the errors made by FAIRR are more interpretable than ProofWriter, since we can pin-point which module is at fault. Whereas, in ProofWriter, it is sometimes hard to understand the source of errors. This feature also makes our framework easier to debug to potentially fix some components with techniques like data augmentation. Please refer to Appendix I for more discussion and examples of errors. A key goal of FAIRR is to explicitly ensure causality from the rule/facts selection step (proof generation) to the reasoning step (intermediate inference generation). This is essential for a reasoning method using forward chaining to solve a deductive reasoning task 4 . To understand if ProofWriter, which uses forward chaining, implicitly does this select-then-reason within the model, we perform the following case study: We sample theories from our subject perturbation dataset where ProofWriter made errors, and manually evaluate the model on inputs with all irrelevant rules/facts deleted. Next we sequentially start adding back the deleted rules/facts to see if the output still remains valid. As shown in Table 4, we see that ProofWriter generates a correct inference for the first row which uses just the essential part of the theory required to generate the conclusion, and starts making errors as more sentences are included. Some more examples are shown in Table 16 in Appendix. This shows that internally ProofWriter is unable to faithfully perform the select-then-reason steps for larger theories. In contrast, FAIRR explicitly separates these steps, leading to a faithful reasoning model. Reasoning in Text Reasoning in text is a well studied problem in NLP. Natural Language Inference (NLI) (Dagan et al., 2006) is one of the most prominent tasks that require reasoning over text to answer if a statement is entailed, contradicted, or neutral, given a hypothesis. More recently, datasets like HotpotQA (Yang et al., 2018), bAbI (Weston et al., 2016), QuaRTz (Tafjord et al., 2019), ROPES (Lin et al., 2019), CLUTRR (Sinha et al., 2019), etc., have studied different aspects of reasoning over textual inputs. These tasks usually require implicit reasoning, where the model needs to internally infer the rules required to solve the task. In contrast, RuleTaker (Clark et al., 2020) deals with explicit reasoning (also known as deductive reasoning). Proof Generation Recently, some works have been addressing the problem of proof generation from an NL-based theory. Prover (Saha et al., 2020) trains a RoBERTa-based model that predicts nodes and edges of the proof graph. ProofWriter (Tafjord et al., 2021) is a T5-based (Raffel et al., 2020) model, that iteratively generates one-hop conclusions and proofs from a theory. Another work MultiProver (Saha et al., 2021), generates multiple possible proofs for a statement. While we study the same problem of proof generation similar to these 4 Forward chaining is described as repeated application of modus ponens (Hinkelmann, 2004), which requires at least two premises to then logically conclude an inference. works, we develop a more faithful and robust model designing a modular system for proof generation. Formal Reasoning There are some prior works that try to solve the problem of entailment prediction by first parsing the formal language from text. Neural Theorem Prover (Rocktschel and Riedel, 2017; Weber et al., 2019) uses neural networks to parse the formal logic from natural language and then reason over them. While this approach is more symbolic, it can lead to many challenges while parsing (Kamath and Das, 2019). The proof generation setting considered here bypasses this step and directly reasons over the given natural language text making it more useful in downstream applications. Model Interpretability With the advent of pre-trained language models (BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), etc.), there has been an increasing trend on solving various reasoning tasks with high accuracy. Faithfulness of such models (Jacovi and Goldberg, 2020) aims to understand whether the models are actually learning to solve the task or rather depending on some shortcut patterns. Saliency-based explanations (Sundarara-jan et al., 2017; Lundberg and Lee, 2017; Murdoch et al., 2018; Sanyal and Ren, 2021) mainly focus on identifying the important phrases in the input text that helped the model in solving a task. In contrast, the task of proof generation focuses on generating a deductive chain of reasoning from the given theory to the concluded statement. Thus, proof chains are easier to understand for end users, making it more useful to debug any systematic model errors. Causal Reasoning The study of causality and causal reasoning models (Pearl, 2000, 2004; Schlkopf, 2019) has been prevalent in machine learning. It has been applied in various domains such as algorithmic fairness (Loftus et al., 2018), gender bias mitigation (Vig et al., 2020), robustness from spurious correlations (Bhlmann, 2020; Veitch et al., 2021), counterfactual explanations (Feder et al., 2021b), etc. Causality in NLP is particularly important to learn models that go beyond exploiting correlations and to improve their overall faithfulness (Feder et al., 2021a). In this paper, we proposed FAIRR, a faithful and robust deductive reasoning model based on three modular components: rule selection, fact selection, and knowledge composition. FAIRR ensures causality from proof generation to entailment prediction by design. We established the effectiveness of our approach through experiments on testing robustness to language variations and demonstrating the interpretability of the errors made by our model. We also show that FAIRR is faster and more precise at deductive reasoning than prior baselines. This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, the DARPA MCS program under Contract No. N660011924033, the Defense Advanced Research Projects Agency with award W911NF-19-20271, NSF IIS 2048211, NSF SMA 1829268, and gift awards from Google, Amazon, JP Morgan and Sony. We would like to thank all the collaborators in USC INK research lab for their constructive feedback on the work.
abstain abstain abstain method abstain abstain objective objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain abstain objective abstain abstain abstain method abstain objective method method method objective abstain result objective abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain method method method abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method method abstain abstain abstain abstain method method abstain method method abstain method abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain result result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other other other other other other other other other method objective other other other other other other other other other other other other other objective abstain objective result other other other